[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:02 dist-test-slave-5c7l krb5kdc[9216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:02 dist-test-slave-5c7l krb5kdc[9216](info): set up 2 sockets
Sep 24 01:02:02 dist-test-slave-5c7l krb5kdc[9216](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 01:02:03.348292 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:03 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:03.528218 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45609
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:03.876899 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:03.877211 9229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:03.877305 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:03.885193 9229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:03.885320 9229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:03.885368 9229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:03.885427 9229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:03.885474 9229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:03.895642 9229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:03.896966 9229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:03.898272 9229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:03.955626 9234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:03.955569 9237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:03.955554 9235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:03.956502 9229 server_base.cc:1034] running on GCE node
I20260924 01:02:03.957427 9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:03.959029 9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:03.960238 9229 hybrid_clock.cc:648] HybridClock initialized: now 1790211723960190 us; error 36 us; skew 500 ppm
I20260924 01:02:03.960641 9229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:03 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:03.963881 9229 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:03.965857 9229 webserver.cc:460] Webserver started at http://127.8.254.126:34149/ using document root <none> and password file <none>
I20260924 01:02:03.966825 9229 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:03.967044 9229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:03.967453 9229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:03.970697 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "70966db2ede74e84b27d6b546c3d268d"
format_stamp: "Formatted at 2026-09-24 01:02:03 on dist-test-slave-5c7l"
server_key: "0abf4942f5268ed98af793ead998cd3e"
server_key_iv: "5ca17c65d017825bfbbaee00b959b68a"
server_key_version: "encryptionkey@0"
I20260924 01:02:03.971606 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "70966db2ede74e84b27d6b546c3d268d"
format_stamp: "Formatted at 2026-09-24 01:02:03 on dist-test-slave-5c7l"
server_key: "0abf4942f5268ed98af793ead998cd3e"
server_key_iv: "5ca17c65d017825bfbbaee00b959b68a"
server_key_version: "encryptionkey@0"
I20260924 01:02:03.977240 9229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:03.981389 9244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:03.982860 9229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:03.983054 9229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "70966db2ede74e84b27d6b546c3d268d"
format_stamp: "Formatted at 2026-09-24 01:02:03 on dist-test-slave-5c7l"
server_key: "0abf4942f5268ed98af793ead998cd3e"
server_key_iv: "5ca17c65d017825bfbbaee00b959b68a"
server_key_version: "encryptionkey@0"
I20260924 01:02:03.983276 9229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:04.008139 9229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:04.012534 9229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:04.012859 9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:04.033910 9229 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45609
I20260924 01:02:04.033921 9295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45609 every 8 connection(s)
I20260924 01:02:04.035378 9229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:04.036840 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9229
I20260924 01:02:04.037328 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:04.039268 9209 external_mini_cluster.cc:1428] Setting key 20956368df0ca4f3a0ddb9c0f3b2e714
I20260924 01:02:04.039842 9296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:04.052380 9296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d: Bootstrap starting.
I20260924 01:02:04.056080 9296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:04.057201 9296 log.cc:824] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:04.060295 9296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d: No bootstrap required, opened a new log
I20260924 01:02:04.068774 9296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70966db2ede74e84b27d6b546c3d268d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45609 } }
I20260924 01:02:04.069175 9296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:04.069295 9296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70966db2ede74e84b27d6b546c3d268d, State: Initialized, Role: FOLLOWER
Sep 24 01:02:04 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:04.070080 9296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70966db2ede74e84b27d6b546c3d268d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45609 } }
I20260924 01:02:04.070397 9296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:04.070565 9296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:04.070756 9296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:04.073757 9296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70966db2ede74e84b27d6b546c3d268d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45609 } }
I20260924 01:02:04.074477 9296 leader_election.cc:302] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70966db2ede74e84b27d6b546c3d268d; no voters:
I20260924 01:02:04.075472 9296 leader_election.cc:288] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:04.075701 9301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:04.077952 9301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [term 1 LEADER]: Becoming Leader. State: Replica: 70966db2ede74e84b27d6b546c3d268d, State: Running, Role: LEADER
I20260924 01:02:04.078495 9296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:04.078612 9301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70966db2ede74e84b27d6b546c3d268d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45609 } }
I20260924 01:02:04.082739 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:04.043274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.8.254.126:45609)
0924 01:02:04.045570 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:04.045586 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:04.045641 (+ 55us) server_negotiation.cc:407] Connection header received
0924 01:02:04.055469 (+ 9828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:04.055491 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:04.055866 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:04.056269 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:04.059599 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:04.061556 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:04.063420 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:04.064395 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:04.064689 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:04.071180 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:04.071189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:04.071202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:04.071240 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:04.074783 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:04.076239 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:04.076256 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:04.076269 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:04.076652 (+ 383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:04.077376 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:04.077384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:04.077390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:04.078362 (+ 972us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:04.078701 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:04.079928 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0924 01:02:04.080452 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057,"thread_start_us":1736,"threads_started":1}
I20260924 01:02:04.083390 9302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70966db2ede74e84b27d6b546c3d268d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70966db2ede74e84b27d6b546c3d268d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45609 } } }
I20260924 01:02:04.083792 9302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:04.084265 9303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70966db2ede74e84b27d6b546c3d268d. Latest consensus state: current_term: 1 leader_uuid: "70966db2ede74e84b27d6b546c3d268d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70966db2ede74e84b27d6b546c3d268d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45609 } } }
I20260924 01:02:04.084546 9303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:04.086740 9306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:04.092581 9306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:04.099334 9318 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 01:02:04.102893 9306 catalog_manager.cc:1347] Generated new cluster ID: feeb01fc12db4403ba651156b5ec3299
I20260924 01:02:04.103018 9306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:04.122915 9306 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:04.124902 9306 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:04.135552 9306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70966db2ede74e84b27d6b546c3d268d: Generated new TSK 0
I20260924 01:02:04.136716 9306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:04.194751 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45609
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:04.566015 9322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:04.566345 9322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:04.566547 9322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:04.573891 9322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:04.574051 9322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:04.574275 9322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:04.584415 9322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:04.585783 9322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:04.587430 9322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:04.601886 9327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:04.601816 9330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:04.601853 9328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:04.602752 9322 server_base.cc:1034] running on GCE node
I20260924 01:02:04.603425 9322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:04.604413 9322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:04.605613 9322 hybrid_clock.cc:648] HybridClock initialized: now 1790211724605567 us; error 40 us; skew 500 ppm
I20260924 01:02:04.605937 9322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:04 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:04.609030 9322 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:04.610474 9322 webserver.cc:460] Webserver started at http://127.8.254.65:37875/ using document root <none> and password file <none>
I20260924 01:02:04.611142 9322 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:04.611281 9322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:04.611603 9322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:04.614575 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "de5d4e4aa6e04ec28f22728db54b8d12"
format_stamp: "Formatted at 2026-09-24 01:02:04 on dist-test-slave-5c7l"
server_key: "cfc9cf605c03ae70ccc13f26deee9c56"
server_key_iv: "da92aee8911e5afb53512d0c56a52282"
server_key_version: "encryptionkey@0"
I20260924 01:02:04.615345 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "de5d4e4aa6e04ec28f22728db54b8d12"
format_stamp: "Formatted at 2026-09-24 01:02:04 on dist-test-slave-5c7l"
server_key: "cfc9cf605c03ae70ccc13f26deee9c56"
server_key_iv: "da92aee8911e5afb53512d0c56a52282"
server_key_version: "encryptionkey@0"
I20260924 01:02:04.620384 9322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:04.624364 9337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:04.625882 9322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:02:04.626129 9322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "de5d4e4aa6e04ec28f22728db54b8d12"
format_stamp: "Formatted at 2026-09-24 01:02:04 on dist-test-slave-5c7l"
server_key: "cfc9cf605c03ae70ccc13f26deee9c56"
server_key_iv: "da92aee8911e5afb53512d0c56a52282"
server_key_version: "encryptionkey@0"
I20260924 01:02:04.626487 9322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:04.642485 9322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:04.647063 9322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:04.647367 9322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:04.648331 9322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:04.649705 9322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:04.649780 9322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:04.649946 9322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:04.650008 9322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:04.689711 9322 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:44117
I20260924 01:02:04.689718 9449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:44117 every 8 connection(s)
I20260924 01:02:04.691295 9322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:04.694638 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9322
I20260924 01:02:04.694933 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:04.695749 9209 external_mini_cluster.cc:1428] Setting key e5e3e54a7629845ae6eb150cf4c4b67c
Sep 24 01:02:04 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:02:04.717100 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:04.694568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:35799 (local address 127.8.254.126:45609)
0924 01:02:04.695221 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:04.695228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:04.696221 (+ 993us) server_negotiation.cc:407] Connection header received
0924 01:02:04.699171 (+ 2950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:04.699175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:04.699241 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:04.699333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:04.702106 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:04.702957 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:04.704965 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:04.705629 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:04.705786 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:04.710976 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:04.710982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:04.710985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:04.711020 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:04.713822 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:04.714667 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:04.714683 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:04.714691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:04.714810 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:04.715279 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:04.715284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:04.715287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:04.715601 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:04.715773 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:04.716500 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 01:02:04.716738 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":285,"threads_started":1}
I20260924 01:02:04.718169 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:04.695155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45609 (local address 127.8.254.65:35799)
0924 01:02:04.696041 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:04.696092 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:04.698774 (+ 2682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:04.699648 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:04.699665 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:04.700579 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:04.701880 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:04.701911 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:04.703201 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:04.703206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:04.704795 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:04.704808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:04.705907 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:04.705917 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:04.706284 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:04.707007 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:04.707083 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:04.710657 (+ 3574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:04.713983 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:04.713994 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:04.714014 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:04.714536 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:04.714937 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:04.714942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:04.714946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:04.715143 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:04.715744 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:04.715753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:04.716109 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 01:02:04.716384 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 01:02:04.716922 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":310,"threads_started":1}
I20260924 01:02:04.719937 9450 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45609
I20260924 01:02:04.720283 9450 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:04.721027 9450 heartbeater.cc:511] Master 127.8.254.126:45609 requested a full tablet report, sending...
I20260924 01:02:04.723589 9261 ts_manager.cc:194] Registered new tserver with Master: de5d4e4aa6e04ec28f22728db54b8d12 (127.8.254.65:44117)
I20260924 01:02:04.725894 9261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:35799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:04.737962 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45609
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:05.086711 9456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:05.087047 9456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:05.087241 9456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:05.094714 9456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:05.094866 9456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:05.095062 9456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:05.105114 9456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:05.106537 9456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:05.107859 9456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:05.122690 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:05.122799 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:05.122823 9464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:05.123656 9456 server_base.cc:1034] running on GCE node
I20260924 01:02:05.124320 9456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:05.125367 9456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:05.126557 9456 hybrid_clock.cc:648] HybridClock initialized: now 1790211725126499 us; error 62 us; skew 500 ppm
I20260924 01:02:05.126894 9456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:05.130039 9456 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:05.131578 9456 webserver.cc:460] Webserver started at http://127.8.254.66:39131/ using document root <none> and password file <none>
I20260924 01:02:05.132232 9456 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:05.132333 9456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:05.132668 9456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:05.135740 9456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "d27062f7c1c04873af79ef09a5de70d9"
format_stamp: "Formatted at 2026-09-24 01:02:05 on dist-test-slave-5c7l"
server_key: "c67801b90d408dfc9e4aaa45cc511932"
server_key_iv: "93742cb314353da95a414e86ea624665"
server_key_version: "encryptionkey@0"
I20260924 01:02:05.136463 9456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "d27062f7c1c04873af79ef09a5de70d9"
format_stamp: "Formatted at 2026-09-24 01:02:05 on dist-test-slave-5c7l"
server_key: "c67801b90d408dfc9e4aaa45cc511932"
server_key_iv: "93742cb314353da95a414e86ea624665"
server_key_version: "encryptionkey@0"
I20260924 01:02:05.141656 9456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:02:05.145217 9471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:05.146653 9456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 01:02:05.146790 9456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "d27062f7c1c04873af79ef09a5de70d9"
format_stamp: "Formatted at 2026-09-24 01:02:05 on dist-test-slave-5c7l"
server_key: "c67801b90d408dfc9e4aaa45cc511932"
server_key_iv: "93742cb314353da95a414e86ea624665"
server_key_version: "encryptionkey@0"
I20260924 01:02:05.146965 9456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:05.171826 9456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:05.176421 9456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:05.176679 9456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:05.177729 9456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:05.179042 9456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:05.179117 9456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:05.179260 9456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:05.179312 9456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:05.221329 9456 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:37373
I20260924 01:02:05.221414 9583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:37373 every 8 connection(s)
I20260924 01:02:05.222697 9456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:05.225669 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9456
I20260924 01:02:05.225885 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:05.226430 9209 external_mini_cluster.cc:1428] Setting key ec522b93276aa7d6b460806fe67b3318
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:02:05.243978 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.225626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:34539 (local address 127.8.254.126:45609)
0924 01:02:05.226162 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:05.226168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:05.227627 (+ 1459us) server_negotiation.cc:407] Connection header received
0924 01:02:05.229171 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:05.229178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:05.229273 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:05.229392 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:05.231132 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.232375 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.233970 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.234490 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.234634 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.238115 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:05.238120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:05.238124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:05.238156 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:05.240872 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.241609 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.241616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.241620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.241752 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.242215 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.242222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.242226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.242501 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:05.242661 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:05.243358 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 01:02:05.243588 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":268,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:05.245113 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.226240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45609 (local address 127.8.254.66:34539)
0924 01:02:05.227088 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:05.227137 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:05.228901 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:05.229741 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:05.229749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:05.230370 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:05.230960 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:05.230979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.232554 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.232558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:05.233814 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:05.233827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.234736 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.234744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:05.234976 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.235693 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:05.235717 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:05.237873 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:05.241017 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:05.241027 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:05.241044 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:05.241482 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:05.241867 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:05.241872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:05.241876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:05.242071 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:05.242636 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:05.242644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:05.243003 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 01:02:05.243279 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 01:02:05.243835 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":349,"threads_started":1}
I20260924 01:02:05.246800 9584 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45609
I20260924 01:02:05.247157 9584 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:05.248046 9584 heartbeater.cc:511] Master 127.8.254.126:45609 requested a full tablet report, sending...
I20260924 01:02:05.249681 9261 ts_manager.cc:194] Registered new tserver with Master: d27062f7c1c04873af79ef09a5de70d9 (127.8.254.66:37373)
I20260924 01:02:05.250597 9261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:34539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:05.266171 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45609
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:05.612717 9590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:05.613020 9590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:05.613119 9590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:05.620590 9590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:05.620699 9590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:05.620816 9590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:05.631313 9590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:05.632705 9590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:05.634029 9590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:05.648016 9596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:05.648514 9598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:05.648736 9595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:05.649364 9590 server_base.cc:1034] running on GCE node
I20260924 01:02:05.649993 9590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:05.651117 9590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:05.652354 9590 hybrid_clock.cc:648] HybridClock initialized: now 1790211725652296 us; error 48 us; skew 500 ppm
I20260924 01:02:05.652705 9590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:05.656657 9590 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:05.658556 9590 webserver.cc:460] Webserver started at http://127.8.254.67:33047/ using document root <none> and password file <none>
I20260924 01:02:05.659263 9590 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:05.659400 9590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:05.659735 9590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:05.662775 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "43099d178d7940e78c1b7a8057659863"
format_stamp: "Formatted at 2026-09-24 01:02:05 on dist-test-slave-5c7l"
server_key: "bf692461bb18f7f4984edc256236259d"
server_key_iv: "b2659ab4b336b1d3fa1b186b817998ee"
server_key_version: "encryptionkey@0"
I20260924 01:02:05.663549 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "43099d178d7940e78c1b7a8057659863"
format_stamp: "Formatted at 2026-09-24 01:02:05 on dist-test-slave-5c7l"
server_key: "bf692461bb18f7f4984edc256236259d"
server_key_iv: "b2659ab4b336b1d3fa1b186b817998ee"
server_key_version: "encryptionkey@0"
I20260924 01:02:05.668919 9590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:02:05.672613 9605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:05.674104 9590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:05.674319 9590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "43099d178d7940e78c1b7a8057659863"
format_stamp: "Formatted at 2026-09-24 01:02:05 on dist-test-slave-5c7l"
server_key: "bf692461bb18f7f4984edc256236259d"
server_key_iv: "b2659ab4b336b1d3fa1b186b817998ee"
server_key_version: "encryptionkey@0"
I20260924 01:02:05.674682 9590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:05.692737 9590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:05.698904 9590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:05.699185 9590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:05.700135 9590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:05.701750 9590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:05.701864 9590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:05.702033 9590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:05.702129 9590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:05.729370 9450 heartbeater.cc:503] Master 127.8.254.126:45609 was elected leader, sending a full tablet report...
I20260924 01:02:05.741745 9590 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:33843
I20260924 01:02:05.741782 9717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:33843 every 8 connection(s)
I20260924 01:02:05.743228 9590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:05.752738 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9590
I20260924 01:02:05.752957 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:05.753523 9209 external_mini_cluster.cc:1428] Setting key 95430e4b9132dddeb264f60f481c0fb7
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:05.764546 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.745782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:47515 (local address 127.8.254.126:45609)
0924 01:02:05.746271 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:05.746277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:05.747332 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 01:02:05.749134 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:05.749139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:05.749219 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:05.749310 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:05.750899 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.752200 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.754654 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.755219 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.755368 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.759034 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:05.759039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:05.759042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:05.759086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:05.761516 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.762287 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.762292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.762296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.762424 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.762831 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.762834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.762836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.763104 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:05.763221 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:05.764043 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 01:02:05.764242 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":203,"threads_started":1}
I20260924 01:02:05.765401 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.746478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45609 (local address 127.8.254.67:47515)
0924 01:02:05.747183 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:05.747219 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:05.748863 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:05.749533 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:05.749541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:05.750090 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:05.750707 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:05.750724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.752382 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.752389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:05.754461 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:05.754474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.755434 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.755440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:05.755662 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.756397 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:05.756419 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:05.758808 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:05.761654 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:05.761661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:05.761675 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:05.762103 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:05.762520 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:05.762523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:05.762526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:05.762729 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:05.763208 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:05.763214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:05.763553 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 01:02:05.763820 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 01:02:05.764232 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":263,"threads_started":1}
I20260924 01:02:05.766918 9718 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45609
I20260924 01:02:05.767232 9718 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:05.768039 9718 heartbeater.cc:511] Master 127.8.254.126:45609 requested a full tablet report, sending...
I20260924 01:02:05.770010 9260 ts_manager.cc:194] Registered new tserver with Master: 43099d178d7940e78c1b7a8057659863 (127.8.254.67:33843)
I20260924 01:02:05.771224 9260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:47515
I20260924 01:02:05.782222 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:05.830049 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.812820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.8.254.65:44117)
0924 01:02:05.813480 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:05.813487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:05.813515 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:02:05.813627 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:05.813634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:05.813850 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:05.813980 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:05.815711 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.816737 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.818301 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.818921 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.819021 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.823495 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:05.823510 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:05.823525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:05.823563 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:05.827110 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.827908 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.827916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.827918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.828068 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.828525 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.828528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.828530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.828888 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:05.829136 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:05.829401 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 01:02:05.829632 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":355,"threads_started":1}
W20260924 01:02:05.831393 9365 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57632
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:05.874462 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.861483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.8.254.65:44117)
0924 01:02:05.861723 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:05.861728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:05.861810 (+ 82us) server_negotiation.cc:407] Connection header received
0924 01:02:05.861963 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:05.861967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:05.862034 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:05.862121 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:05.863473 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.864131 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.865266 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.865681 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.865773 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.869454 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:05.869459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:05.869463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:05.869496 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:05.872115 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.872720 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.872724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.872727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.872852 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.873172 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.873191 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.873196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.873457 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:05.873642 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:05.873912 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 01:02:05.874079 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 01:02:05.875984 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9322
I20260924 01:02:05.889356 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.876945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.8.254.65:44117)
0924 01:02:05.877164 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:05.877172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:05.877483 (+ 311us) server_negotiation.cc:407] Connection header received
0924 01:02:05.877593 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:05.877596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:05.877654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:05.877750 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:05.879139 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.879850 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.881180 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.881580 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.881672 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:05.883909 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:05.883914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:05.883917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:05.883945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:05.886934 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.887603 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.887606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.887609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.887744 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:05.888112 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:05.888115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:05.888117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:05.888355 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:05.888532 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:05.888814 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 01:02:05.889015 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 01:02:05.890179 9445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:05.972862 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9322
I20260924 01:02:05.990515 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9456
Sep 24 01:02:05 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:06.007072 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:05.991562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.8.254.66:37373)
0924 01:02:05.992177 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:05.992184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:05.992210 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:02:05.992273 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:05.992280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:05.992490 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:05.992626 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:05.993789 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.994950 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.996147 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:05.996750 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:05.996861 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:06.001193 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:06.001208 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:06.001224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:06.001274 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:06.004285 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:06.004912 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:06.004919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:06.004921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:06.005090 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:06.005435 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:06.005438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:06.005440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:06.005782 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:06.006055 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:06.006370 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 01:02:06.006647 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":324,"threads_started":1}
I20260924 01:02:06.008441 9579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:06.098352 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9456
I20260924 01:02:06.121223 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9590
Sep 24 01:02:06 dist-test-slave-5c7l krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:06.137096 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:06.122343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.8.254.67:33843)
0924 01:02:06.122993 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:06.123000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:06.123029 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:02:06.123087 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:06.123093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:06.123307 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:06.123439 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:06.124603 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:06.125615 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:06.127004 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:06.127550 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:06.127648 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:06.131796 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:06.131808 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:06.131824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:06.131858 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:06.134628 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:06.135319 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:06.135326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:06.135328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:06.135454 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:06.135771 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:06.135774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:06.135776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:06.136116 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:06.136363 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:06.136602 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 01:02:06.136815 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":361,"threads_started":1}
I20260924 01:02:06.138445 9713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:06.221127 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9590
I20260924 01:02:06.238161 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9229
I20260924 01:02:06.239426 9291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:06.346118 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9229
2026-09-24T01:02:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4098 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:06 dist-test-slave-5c7l krb5kdc[9740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:06 dist-test-slave-5c7l krb5kdc[9740](info): set up 2 sockets
Sep 24 01:02:06 dist-test-slave-5c7l krb5kdc[9740](info): commencing operation
krb5kdc: starting...
W20260924 01:02:07.431335 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:07 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:07.613760 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38849
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:07.933917 9753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:07.934312 9753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:07.934469 9753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:07.941763 9753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:07.941915 9753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:07.942025 9753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:07.942119 9753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:07.942199 9753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:07.952211 9753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:07.953681 9753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:07.955081 9753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:07.966979 9761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:07.967070 9759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:07.967015 9758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:07.967862 9753 server_base.cc:1034] running on GCE node
I20260924 01:02:07.968714 9753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:07.970170 9753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:07.971441 9753 hybrid_clock.cc:648] HybridClock initialized: now 1790211727971387 us; error 42 us; skew 500 ppm
I20260924 01:02:07.971808 9753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:07 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:07.988646 9753 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:07.990550 9753 webserver.cc:460] Webserver started at http://127.8.254.126:41731/ using document root <none> and password file <none>
I20260924 01:02:07.991323 9753 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:07.991473 9753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:07.991815 9753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:07.995019 9753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "0c730225991944cdadd243d051c8fcd0"
format_stamp: "Formatted at 2026-09-24 01:02:07 on dist-test-slave-5c7l"
server_key: "be9d8cec7128aa2afaf2757dc0f15d01"
server_key_iv: "d0518bc271b4c96f9689aeed7ae343f9"
server_key_version: "encryptionkey@0"
I20260924 01:02:07.995832 9753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "0c730225991944cdadd243d051c8fcd0"
format_stamp: "Formatted at 2026-09-24 01:02:07 on dist-test-slave-5c7l"
server_key: "be9d8cec7128aa2afaf2757dc0f15d01"
server_key_iv: "d0518bc271b4c96f9689aeed7ae343f9"
server_key_version: "encryptionkey@0"
I20260924 01:02:08.001493 9753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:02:08.005511 9768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:08.006958 9753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:08.007157 9753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "0c730225991944cdadd243d051c8fcd0"
format_stamp: "Formatted at 2026-09-24 01:02:07 on dist-test-slave-5c7l"
server_key: "be9d8cec7128aa2afaf2757dc0f15d01"
server_key_iv: "d0518bc271b4c96f9689aeed7ae343f9"
server_key_version: "encryptionkey@0"
I20260924 01:02:08.007377 9753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:08.030474 9753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:08.034837 9753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:08.035153 9753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:08.054163 9753 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:38849
I20260924 01:02:08.054309 9819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:38849 every 8 connection(s)
I20260924 01:02:08.055653 9753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:08.059504 9820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:08.061290 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9753
I20260924 01:02:08.061533 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:08.062376 9209 external_mini_cluster.cc:1428] Setting key 94b7a6c65b028000d0d85f57eadb772b
I20260924 01:02:08.074363 9820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0: Bootstrap starting.
Sep 24 01:02:08 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:08.078330 9820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:08.079422 9820 log.cc:824] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:08.082384 9820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0: No bootstrap required, opened a new log
I20260924 01:02:08.085788 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:08.064510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.8.254.126:38849)
0924 01:02:08.065336 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:08.065348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:08.065394 (+ 46us) server_negotiation.cc:407] Connection header received
0924 01:02:08.067052 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:08.067081 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:08.067491 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:08.067832 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:08.069646 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:08.070997 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:08.072811 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:08.073480 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:08.073624 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:08.078341 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:08.078352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:08.078367 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:08.078406 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:08.081583 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:08.082335 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:08.082350 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:08.082356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:08.082559 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:08.082944 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:08.082950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:08.082953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:08.083475 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:08.083803 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:08.084331 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 01:02:08.084770 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":396,"threads_started":1}
I20260924 01:02:08.090436 9820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c730225991944cdadd243d051c8fcd0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38849 } }
I20260924 01:02:08.090852 9820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:08.090958 9820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c730225991944cdadd243d051c8fcd0, State: Initialized, Role: FOLLOWER
I20260924 01:02:08.091506 9820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c730225991944cdadd243d051c8fcd0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38849 } }
I20260924 01:02:08.091753 9820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:08.091897 9820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:08.092067 9820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:08.094457 9820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c730225991944cdadd243d051c8fcd0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38849 } }
I20260924 01:02:08.094993 9820 leader_election.cc:302] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c730225991944cdadd243d051c8fcd0; no voters:
I20260924 01:02:08.095630 9820 leader_election.cc:288] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:08.095731 9825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:08.096472 9825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 0c730225991944cdadd243d051c8fcd0, State: Running, Role: LEADER
I20260924 01:02:08.097080 9825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c730225991944cdadd243d051c8fcd0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38849 } }
I20260924 01:02:08.097805 9820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:08.099714 9826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c730225991944cdadd243d051c8fcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c730225991944cdadd243d051c8fcd0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38849 } } }
I20260924 01:02:08.100142 9826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:08.100908 9827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c730225991944cdadd243d051c8fcd0. Latest consensus state: current_term: 1 leader_uuid: "0c730225991944cdadd243d051c8fcd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c730225991944cdadd243d051c8fcd0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38849 } } }
I20260924 01:02:08.101308 9827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:08.102015 9832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:08.107157 9832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:08.116247 9832 catalog_manager.cc:1347] Generated new cluster ID: 724428f674584ca68abcb804529accf5
I20260924 01:02:08.116359 9832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:08.129062 9832 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:08.130568 9832 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:08.137778 9832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c730225991944cdadd243d051c8fcd0: Generated new TSK 0
I20260924 01:02:08.138661 9832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:08.190410 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38849
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:08.536092 9846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:08.536402 9846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:08.536576 9846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:08.543857 9846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:08.544019 9846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:08.544215 9846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:08.553984 9846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:08.555354 9846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:08.556710 9846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:08.569805 9854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:08.569746 9851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:08.569892 9852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:08.571012 9846 server_base.cc:1034] running on GCE node
I20260924 01:02:08.571640 9846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:08.572615 9846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:08.573845 9846 hybrid_clock.cc:648] HybridClock initialized: now 1790211728573788 us; error 36 us; skew 500 ppm
I20260924 01:02:08.574337 9846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:08 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211728, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:08.577585 9846 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:08.579180 9846 webserver.cc:460] Webserver started at http://127.8.254.65:34675/ using document root <none> and password file <none>
I20260924 01:02:08.579898 9846 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:08.580099 9846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:08.580420 9846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:08.583459 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "82422cae27e546eeb7a4a844f69d06a5"
format_stamp: "Formatted at 2026-09-24 01:02:08 on dist-test-slave-5c7l"
server_key: "8cd563f130f3ffed915cca0584dec4d5"
server_key_iv: "de146a1689f737c411b3b4d036df65a7"
server_key_version: "encryptionkey@0"
I20260924 01:02:08.584344 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "82422cae27e546eeb7a4a844f69d06a5"
format_stamp: "Formatted at 2026-09-24 01:02:08 on dist-test-slave-5c7l"
server_key: "8cd563f130f3ffed915cca0584dec4d5"
server_key_iv: "de146a1689f737c411b3b4d036df65a7"
server_key_version: "encryptionkey@0"
I20260924 01:02:08.589777 9846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:08.593475 9861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:08.594945 9846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 01:02:08.595129 9846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "82422cae27e546eeb7a4a844f69d06a5"
format_stamp: "Formatted at 2026-09-24 01:02:08 on dist-test-slave-5c7l"
server_key: "8cd563f130f3ffed915cca0584dec4d5"
server_key_iv: "de146a1689f737c411b3b4d036df65a7"
server_key_version: "encryptionkey@0"
I20260924 01:02:08.595336 9846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:08.632156 9846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:08.636483 9846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:08.636859 9846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:08.637948 9846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:08.639447 9846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:08.639520 9846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:08.639696 9846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:08.639796 9846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:08.672125 9846 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:35109
I20260924 01:02:08.672174 9973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:35109 every 8 connection(s)
I20260924 01:02:08.673514 9846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:08.680433 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9846
I20260924 01:02:08.680670 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:08.681309 9209 external_mini_cluster.cc:1428] Setting key a6ff49db1ad9d5c7bb76e02faef4eeff
Sep 24 01:02:08 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211728, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:08.695804 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:08.676177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:34219 (local address 127.8.254.126:38849)
0924 01:02:08.676649 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:08.676656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:08.677679 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 01:02:08.679491 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:08.679496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:08.679574 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:08.679674 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:08.681223 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:08.681990 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:08.684160 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:08.685004 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:08.685184 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:08.689409 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:08.689420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:08.689426 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:08.689489 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:08.692170 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:08.693269 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:08.693273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:08.693275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:08.693391 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:08.693924 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:08.693927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:08.693929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:08.694238 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:08.694371 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:08.695374 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0924 01:02:08.695544 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":215,"threads_started":1}
I20260924 01:02:08.697654 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:08.676831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38849 (local address 127.8.254.65:34219)
0924 01:02:08.677518 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:08.677556 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:08.679244 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:08.679886 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:08.679893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:08.680388 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:08.681042 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:08.681060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:08.682183 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:08.682187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:08.683988 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:08.684000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:08.685337 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:08.685348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:08.685716 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:08.686488 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:08.686516 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:08.689139 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:08.692329 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:08.692343 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:08.692365 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:08.693140 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:08.693504 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:08.693512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:08.693520 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:08.693796 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:08.694386 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:08.694398 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:08.694922 (+ 524us) client_negotiation.cc:769] Sending connection context
0924 01:02:08.695285 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 01:02:08.695925 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":273,"threads_started":1}
I20260924 01:02:08.699635 9974 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38849
I20260924 01:02:08.699975 9974 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:08.700778 9974 heartbeater.cc:511] Master 127.8.254.126:38849 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:02:08.703347 9785 ts_manager.cc:194] Registered new tserver with Master: 82422cae27e546eeb7a4a844f69d06a5 (127.8.254.65:35109)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:02:08.704954 9785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:34219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:08.725968 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38849
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:09.071938 9980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:09.072218 9980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:09.072415 9980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:09.080860 9980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:09.081007 9980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:09.081210 9980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:09.093344 9980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:09.094893 9980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:09.096345 9980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:09.112051 9986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:09.112277 9988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:09.112287 9980 server_base.cc:1034] running on GCE node
W20260924 01:02:09.112653 9985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:09.113385 9980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:09.114377 9980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:09.115586 9980 hybrid_clock.cc:648] HybridClock initialized: now 1790211729115530 us; error 51 us; skew 500 ppm
I20260924 01:02:09.115917 9980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:09 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211729, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:09.119395 9980 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:09.121130 9980 webserver.cc:460] Webserver started at http://127.8.254.66:41947/ using document root <none> and password file <none>
I20260924 01:02:09.121837 9980 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:09.121971 9980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:09.122349 9980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:09.125278 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "678be2d7992e48d79d9c5cd253c93761"
format_stamp: "Formatted at 2026-09-24 01:02:09 on dist-test-slave-5c7l"
server_key: "4402ad11bd1742315521b25d37aa0f62"
server_key_iv: "18e2646710363fc3efbfe0fa90767725"
server_key_version: "encryptionkey@0"
I20260924 01:02:09.126103 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "678be2d7992e48d79d9c5cd253c93761"
format_stamp: "Formatted at 2026-09-24 01:02:09 on dist-test-slave-5c7l"
server_key: "4402ad11bd1742315521b25d37aa0f62"
server_key_iv: "18e2646710363fc3efbfe0fa90767725"
server_key_version: "encryptionkey@0"
I20260924 01:02:09.131498 9980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 01:02:09.135035 9995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:09.136333 9980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:09.136514 9980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "678be2d7992e48d79d9c5cd253c93761"
format_stamp: "Formatted at 2026-09-24 01:02:09 on dist-test-slave-5c7l"
server_key: "4402ad11bd1742315521b25d37aa0f62"
server_key_iv: "18e2646710363fc3efbfe0fa90767725"
server_key_version: "encryptionkey@0"
I20260924 01:02:09.136755 9980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:09.158569 9980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:09.163218 9980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:09.163527 9980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:09.164515 9980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:09.166046 9980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:09.166281 9980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 01:02:09.166510 9980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:09.166671 9980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:09.212518 9980 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:43271
I20260924 01:02:09.212550 10107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:43271 every 8 connection(s)
I20260924 01:02:09.213932 9980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:09.214315 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 9980
I20260924 01:02:09.214515 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:09.215220 9209 external_mini_cluster.cc:1428] Setting key 6e28873b973d681b7f0b98771d802548
Sep 24 01:02:09 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211729, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:09.234761 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.217067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:33815 (local address 127.8.254.126:38849)
0924 01:02:09.217581 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:09.217589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:09.218612 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 01:02:09.220513 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:09.220523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:09.220622 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:09.220750 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:09.222239 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.222939 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.224481 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.224904 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.225042 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.228691 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:09.228697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:09.228706 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:09.228741 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:09.231560 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:09.232384 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:09.232390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:09.232394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:09.232525 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:09.232970 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:09.232975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:09.232979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:09.233264 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:09.233417 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:09.234128 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 01:02:09.234376 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":265,"threads_started":1}
I20260924 01:02:09.236093 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.217608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38849 (local address 127.8.254.66:33815)
0924 01:02:09.218433 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:09.218478 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:09.220235 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:09.220959 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:09.220966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:09.221471 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:09.222031 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.222050 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.223099 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.223105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.224330 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.224343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.225131 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.225138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.225390 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.226083 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:09.226106 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:09.228449 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:09.231712 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:09.231723 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:09.231740 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:09.232259 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:09.232642 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:09.232646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:09.232650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:09.232841 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:09.233407 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:09.233414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:09.233759 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 01:02:09.234040 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 01:02:09.234621 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":336,"threads_started":1}
I20260924 01:02:09.237596 10108 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38849
I20260924 01:02:09.237944 10108 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:09.238868 10108 heartbeater.cc:511] Master 127.8.254.126:38849 requested a full tablet report, sending...
I20260924 01:02:09.240732 9785 ts_manager.cc:194] Registered new tserver with Master: 678be2d7992e48d79d9c5cd253c93761 (127.8.254.66:43271)
I20260924 01:02:09.241623 9785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:33815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:09.259473 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38849
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:09.625377 10114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:09.625658 10114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:09.625829 10114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:09.633260 10114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:09.633380 10114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:09.633591 10114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:09.644042 10114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:09.645656 10114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:09.647207 10114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:09.660750 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:09.660766 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:09.660986 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:09.661494 10114 server_base.cc:1034] running on GCE node
I20260924 01:02:09.662240 10114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:09.663223 10114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:09.664434 10114 hybrid_clock.cc:648] HybridClock initialized: now 1790211729664378 us; error 46 us; skew 500 ppm
I20260924 01:02:09.664740 10114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:09 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211729, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:09.667994 10114 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:09.669409 10114 webserver.cc:460] Webserver started at http://127.8.254.67:41741/ using document root <none> and password file <none>
I20260924 01:02:09.670069 10114 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:09.670166 10114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:09.670498 10114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:09.674535 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "f8d5b4087ed746b0be56e28fc1e71864"
format_stamp: "Formatted at 2026-09-24 01:02:09 on dist-test-slave-5c7l"
server_key: "123b4cf19dcd9afff8521cfd36084874"
server_key_iv: "c311888dc3c14e6e960d5e3e3d370280"
server_key_version: "encryptionkey@0"
I20260924 01:02:09.675611 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "f8d5b4087ed746b0be56e28fc1e71864"
format_stamp: "Formatted at 2026-09-24 01:02:09 on dist-test-slave-5c7l"
server_key: "123b4cf19dcd9afff8521cfd36084874"
server_key_iv: "c311888dc3c14e6e960d5e3e3d370280"
server_key_version: "encryptionkey@0"
I20260924 01:02:09.681529 10114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 01:02:09.685150 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:09.686662 10114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 01:02:09.686798 10114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "f8d5b4087ed746b0be56e28fc1e71864"
format_stamp: "Formatted at 2026-09-24 01:02:09 on dist-test-slave-5c7l"
server_key: "123b4cf19dcd9afff8521cfd36084874"
server_key_iv: "c311888dc3c14e6e960d5e3e3d370280"
server_key_version: "encryptionkey@0"
I20260924 01:02:09.686969 10114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:09.708662 9974 heartbeater.cc:503] Master 127.8.254.126:38849 was elected leader, sending a full tablet report...
I20260924 01:02:09.716446 10114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:09.720845 10114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:09.721148 10114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:09.722144 10114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:09.723547 10114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:09.723623 10114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:09.723785 10114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:09.723837 10114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:09.764540 10114 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:36759
I20260924 01:02:09.764596 10241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:36759 every 8 connection(s)
I20260924 01:02:09.766040 10114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:09.767860 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 10114
I20260924 01:02:09.768081 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:09.768571 9209 external_mini_cluster.cc:1428] Setting key 381166dbb7e7b0d5d27836d71c22625e
Sep 24 01:02:09 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211729, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:09.789710 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.769049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:48925 (local address 127.8.254.126:38849)
0924 01:02:09.769557 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:09.769564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:09.773643 (+ 4079us) server_negotiation.cc:407] Connection header received
0924 01:02:09.773695 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:09.773698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:09.773778 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:09.773861 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:09.775566 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.776659 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.778325 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.778874 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.778977 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.782618 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:09.782625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:09.782629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:09.782673 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:09.786094 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:09.786993 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:09.786998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:09.787001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:09.787126 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:09.787603 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:09.787609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:09.787617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:09.787919 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:09.788094 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:09.788956 (+ 862us) server_negotiation.cc:299] Negotiation successful
0924 01:02:09.789325 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260924 01:02:09.790874 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.769689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38849 (local address 127.8.254.67:48925)
0924 01:02:09.770527 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:09.770574 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:09.773350 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:09.774141 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:09.774150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:09.774800 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:09.775404 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.775422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.776802 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.776805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.778095 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.778108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.779157 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.779165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.779376 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.780033 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:09.780057 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:09.782356 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:09.786303 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:09.786313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:09.786336 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:09.786845 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:09.787252 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:09.787257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:09.787261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:09.787475 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:09.788061 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:09.788070 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:09.788466 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 01:02:09.788750 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 01:02:09.789469 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":355,"threads_started":1}
I20260924 01:02:09.792356 10242 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38849
I20260924 01:02:09.792707 10242 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:09.793491 10242 heartbeater.cc:511] Master 127.8.254.126:38849 requested a full tablet report, sending...
I20260924 01:02:09.795094 9785 ts_manager.cc:194] Registered new tserver with Master: f8d5b4087ed746b0be56e28fc1e71864 (127.8.254.67:36759)
I20260924 01:02:09.795950 9785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:48925
I20260924 01:02:09.797333 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:02:09.818050 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.804382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.8.254.126:38849)
0924 01:02:09.804540 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:09.804545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:09.805023 (+ 478us) server_negotiation.cc:407] Connection header received
0924 01:02:09.805192 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:09.805196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:09.805308 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:09.805418 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:09.807156 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.807934 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.809176 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.809563 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.809659 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.812034 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:09.812041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:09.812047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:09.812086 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:09.815326 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:09.816005 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:09.816010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:09.816014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:09.816145 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:09.816531 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:09.816536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:09.816539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:09.816882 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:09.817142 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:09.817480 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 01:02:09.817718 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 01:02:09.824497 9785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:09.831898 9785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:09.858006 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.845417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.8.254.67:36759)
0924 01:02:09.847998 (+ 2581us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:09.848009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:09.848079 (+ 70us) server_negotiation.cc:407] Connection header received
0924 01:02:09.848193 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:09.848204 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:09.848543 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:09.849687 (+ 1144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:09.849994 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.851822 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.855796 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.856740 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.856898 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.857156 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:09.857451 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 01:02:09.857626 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2319,"thread_start_us":2022,"threads_started":1}
I20260924 01:02:09.858090 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.844660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35109 (local address 127.0.0.1:45592)
0924 01:02:09.845541 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:09.845583 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:09.845759 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:09.849575 (+ 3816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:09.849580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:09.849609 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:09.850088 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.850105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.852813 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.852835 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.855230 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.855242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.857163 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.857170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.857342 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.857384 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 01:02:09.857459 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 01:02:09.857621 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":88,"thread_start_us":1270,"threads_started":1}
I20260924 01:02:09.858587 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.845767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.8.254.66:43271)
0924 01:02:09.847090 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:09.847101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:09.848741 (+ 1640us) server_negotiation.cc:407] Connection header received
0924 01:02:09.849089 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:09.849098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:09.849359 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:09.849563 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:09.850719 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.851896 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.855437 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.856910 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.857072 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.857287 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:09.858042 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 01:02:09.858249 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":2182,"threads_started":1}
I20260924 01:02:09.858937 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.844819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:36759 (local address 127.0.0.1:45266)
0924 01:02:09.846452 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:09.846474 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:09.846584 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:09.848825 (+ 2241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:09.848833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:09.848908 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:09.849446 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.849456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.851995 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.852004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.855642 (+ 3638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.855653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.856960 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.856966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.857175 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.857248 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 01:02:09.857395 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 01:02:09.857616 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"mutex_wait_us":113,"thread_start_us":3479,"threads_started":1}
I20260924 01:02:09.859273 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.845311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43271 (local address 127.0.0.1:45536)
0924 01:02:09.848623 (+ 3312us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:09.848641 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:09.848840 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:09.849845 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:09.849855 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:09.849879 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:09.850490 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.850505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.852122 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.852126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.855235 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:09.855251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.857435 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.857444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:09.857629 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.857656 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 01:02:09.857730 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 01:02:09.857869 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3168,"mutex_wait_us":273,"thread_start_us":1266,"threads_started":1}
I20260924 01:02:09.859937 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:09.844788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.8.254.65:35109)
0924 01:02:09.849057 (+ 4269us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:09.849065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:09.849094 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:02:09.849191 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:09.849197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:09.849359 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:09.849566 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:09.850334 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.852665 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.855391 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:09.856917 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:09.857873 (+ 956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:09.858103 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:09.858172 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 01:02:09.859580 (+ 1408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4113,"thread_start_us":3969,"threads_started":1}
I20260924 01:02:09.862843 10043 tablet_service.cc:1467] Processing CreateTablet for tablet 4a405c7283354ba5b436f80734f1f6f1 (DEFAULT_TABLE table=test-table [id=07a61c0694ca485d9015b28647d62e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:09.863000 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 4a405c7283354ba5b436f80734f1f6f1 (DEFAULT_TABLE table=test-table [id=07a61c0694ca485d9015b28647d62e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:09.863374 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 4a405c7283354ba5b436f80734f1f6f1 (DEFAULT_TABLE table=test-table [id=07a61c0694ca485d9015b28647d62e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:09.864293 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a405c7283354ba5b436f80734f1f6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:09.864457 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a405c7283354ba5b436f80734f1f6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:09.864733 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a405c7283354ba5b436f80734f1f6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:09.875547 10261 tablet_bootstrap.cc:492] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: Bootstrap starting.
I20260924 01:02:09.879136 10262 tablet_bootstrap.cc:492] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: Bootstrap starting.
I20260924 01:02:09.879946 10261 tablet_bootstrap.cc:654] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:09.880924 10263 tablet_bootstrap.cc:492] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: Bootstrap starting.
I20260924 01:02:09.881453 10261 log.cc:824] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:09.882715 10262 tablet_bootstrap.cc:654] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:09.883867 10262 log.cc:824] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:09.884953 10263 tablet_bootstrap.cc:654] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:09.885196 10261 tablet_bootstrap.cc:492] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: No bootstrap required, opened a new log
I20260924 01:02:09.885509 10261 ts_tablet_manager.cc:1397] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260924 01:02:09.886263 10263 log.cc:824] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:09.887596 10262 tablet_bootstrap.cc:492] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: No bootstrap required, opened a new log
I20260924 01:02:09.887852 10262 ts_tablet_manager.cc:1397] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 01:02:09.892674 10263 tablet_bootstrap.cc:492] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: No bootstrap required, opened a new log
I20260924 01:02:09.892949 10263 ts_tablet_manager.cc:1397] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260924 01:02:09.897069 10261 raft_consensus.cc:348] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:09.898167 10261 raft_consensus.cc:374] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:09.898372 10261 raft_consensus.cc:729] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678be2d7992e48d79d9c5cd253c93761, State: Initialized, Role: FOLLOWER
I20260924 01:02:09.898566 10262 raft_consensus.cc:348] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:09.899009 10262 raft_consensus.cc:374] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:09.899116 10262 raft_consensus.cc:729] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82422cae27e546eeb7a4a844f69d06a5, State: Initialized, Role: FOLLOWER
I20260924 01:02:09.899112 10261 consensus_queue.cc:260] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:09.899770 10262 consensus_queue.cc:260] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:09.900754 10108 heartbeater.cc:503] Master 127.8.254.126:38849 was elected leader, sending a full tablet report...
I20260924 01:02:09.901850 10261 ts_tablet_manager.cc:1428] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260924 01:02:09.904495 10263 raft_consensus.cc:348] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:09.904912 10263 raft_consensus.cc:374] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:09.905023 10263 raft_consensus.cc:729] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d5b4087ed746b0be56e28fc1e71864, State: Initialized, Role: FOLLOWER
I20260924 01:02:09.905735 10263 consensus_queue.cc:260] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:09.908691 10262 ts_tablet_manager.cc:1428] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: Time spent starting tablet: real 0.021s user 0.010s sys 0.009s
I20260924 01:02:09.909691 10242 heartbeater.cc:503] Master 127.8.254.126:38849 was elected leader, sending a full tablet report...
I20260924 01:02:09.910593 10263 ts_tablet_manager.cc:1428] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
W20260924 01:02:09.926100 9975 tablet.cc:2369] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:09.967410 10109 tablet.cc:2369] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:10.017803 10243 tablet.cc:2369] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:02:10.220042 10267 raft_consensus.cc:482] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:10.220338 10267 raft_consensus.cc:504] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:10.222548 10267 leader_election.cc:288] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82422cae27e546eeb7a4a844f69d06a5 (127.8.254.65:35109), f8d5b4087ed746b0be56e28fc1e71864 (127.8.254.67:36759)
I20260924 01:02:10.229434 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.223099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:57675 (local address 127.8.254.65:35109)
0924 01:02:10.223299 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.223308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.224083 (+ 775us) server_negotiation.cc:407] Connection header received
0924 01:02:10.224274 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.224279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.224357 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.224461 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:10.225157 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.225921 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.227683 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.228432 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.228627 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.228698 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.229023 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.229153 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:02:10.229456 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.223233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:46547 (local address 127.8.254.67:36759)
0924 01:02:10.223401 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.223405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.223919 (+ 514us) server_negotiation.cc:407] Connection header received
0924 01:02:10.224080 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.224085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.224178 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.224268 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:10.224950 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.225740 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.227644 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.228410 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.228565 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.228655 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.229063 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.229202 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:02:10.229568 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.222914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35109 (local address 127.8.254.66:57675)
0924 01:02:10.223926 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:10.223947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:10.224100 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:10.224486 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:10.224489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:10.224512 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:10.224978 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:10.224992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.226056 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.226060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:10.227487 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:10.227500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.228698 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.228711 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:10.228839 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.228882 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 01:02:10.228957 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 01:02:10.229120 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":286,"thread_start_us":502,"threads_started":1}
I20260924 01:02:10.229612 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.222887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:36759 (local address 127.8.254.66:46547)
0924 01:02:10.223780 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:10.223801 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:10.223933 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:10.224299 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:10.224303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:10.224333 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:10.224786 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:10.224800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.225881 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.225886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:10.227472 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:10.227485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.228646 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.228653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:10.228822 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.228898 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 01:02:10.229008 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 01:02:10.229146 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":95,"thread_start_us":453,"threads_started":1}
I20260924 01:02:10.230736 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a405c7283354ba5b436f80734f1f6f1" candidate_uuid: "678be2d7992e48d79d9c5cd253c93761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82422cae27e546eeb7a4a844f69d06a5" is_pre_election: true
I20260924 01:02:10.230822 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a405c7283354ba5b436f80734f1f6f1" candidate_uuid: "678be2d7992e48d79d9c5cd253c93761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d5b4087ed746b0be56e28fc1e71864" is_pre_election: true
I20260924 01:02:10.231350 9929 raft_consensus.cc:2454] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678be2d7992e48d79d9c5cd253c93761 in term 0.
I20260924 01:02:10.231464 10197 raft_consensus.cc:2454] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678be2d7992e48d79d9c5cd253c93761 in term 0.
I20260924 01:02:10.232257 9996 leader_election.cc:302] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678be2d7992e48d79d9c5cd253c93761, f8d5b4087ed746b0be56e28fc1e71864; no voters:
I20260924 01:02:10.232761 10267 raft_consensus.cc:2781] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:10.232918 10267 raft_consensus.cc:482] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:10.233043 10267 raft_consensus.cc:3037] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:10.235312 10267 raft_consensus.cc:504] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:10.235901 10267 leader_election.cc:288] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [CANDIDATE]: Term 1 election: Requested vote from peers 82422cae27e546eeb7a4a844f69d06a5 (127.8.254.65:35109), f8d5b4087ed746b0be56e28fc1e71864 (127.8.254.67:36759)
I20260924 01:02:10.236405 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a405c7283354ba5b436f80734f1f6f1" candidate_uuid: "678be2d7992e48d79d9c5cd253c93761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d5b4087ed746b0be56e28fc1e71864"
I20260924 01:02:10.236531 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a405c7283354ba5b436f80734f1f6f1" candidate_uuid: "678be2d7992e48d79d9c5cd253c93761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82422cae27e546eeb7a4a844f69d06a5"
I20260924 01:02:10.236605 10197 raft_consensus.cc:3037] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:10.236815 9929 raft_consensus.cc:3037] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:10.238891 10197 raft_consensus.cc:2454] T 4a405c7283354ba5b436f80734f1f6f1 P f8d5b4087ed746b0be56e28fc1e71864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678be2d7992e48d79d9c5cd253c93761 in term 1.
I20260924 01:02:10.239315 9996 leader_election.cc:302] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678be2d7992e48d79d9c5cd253c93761, f8d5b4087ed746b0be56e28fc1e71864; no voters:
I20260924 01:02:10.239619 10267 raft_consensus.cc:2781] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:10.240168 10267 raft_consensus.cc:686] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [term 1 LEADER]: Becoming Leader. State: Replica: 678be2d7992e48d79d9c5cd253c93761, State: Running, Role: LEADER
I20260924 01:02:10.240269 9929 raft_consensus.cc:2454] T 4a405c7283354ba5b436f80734f1f6f1 P 82422cae27e546eeb7a4a844f69d06a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678be2d7992e48d79d9c5cd253c93761 in term 1.
I20260924 01:02:10.241086 10267 consensus_queue.cc:237] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } }
I20260924 01:02:10.245777 9785 catalog_manager.cc:5495] T 4a405c7283354ba5b436f80734f1f6f1 P 678be2d7992e48d79d9c5cd253c93761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 678be2d7992e48d79d9c5cd253c93761 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "678be2d7992e48d79d9c5cd253c93761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82422cae27e546eeb7a4a844f69d06a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8d5b4087ed746b0be56e28fc1e71864" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678be2d7992e48d79d9c5cd253c93761" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43271 } health_report { overall_health: HEALTHY } } }
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:10.320231 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.304113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.8.254.65:35109)
0924 01:02:10.304359 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.304366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.304862 (+ 496us) server_negotiation.cc:407] Connection header received
0924 01:02:10.305055 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.305060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.305141 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.305227 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:10.307436 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.308219 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.309436 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.309848 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.309943 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.314175 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:10.314262 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:10.314287 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:10.314324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:10.317404 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.318159 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.318169 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.318172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.318407 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.318787 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.318790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.318792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.319141 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:10.319367 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.319648 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.319811 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:10.366171 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.351686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.8.254.65:35109)
0924 01:02:10.351897 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.351906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.352237 (+ 331us) server_negotiation.cc:407] Connection header received
0924 01:02:10.352461 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.352465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.352532 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.352636 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:10.353974 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.354790 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.356024 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.356445 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.356579 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.360682 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:10.360688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:10.360692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:10.360731 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:10.363453 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.364140 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.364146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.364151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.364285 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.364716 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.364727 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.364730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.364999 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:10.365300 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.365575 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.365805 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:10.412050 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.397955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.8.254.65:35109)
0924 01:02:10.398193 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.398229 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.398544 (+ 315us) server_negotiation.cc:407] Connection header received
0924 01:02:10.398811 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.398816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.398884 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.399010 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:10.400424 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.401170 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.402516 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.402941 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.403074 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.406908 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:10.406914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:10.406917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:10.406951 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:10.409408 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.410072 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.410076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.410078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.410234 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.410551 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.410554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.410556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.410833 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:10.411017 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.411458 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.411692 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 01:02:10.413736 9889 tablet_service.cc:2908] Rejecting scan request for tablet 4a405c7283354ba5b436f80734f1f6f1: Uninitialized: clean time has not yet been initialized
I20260924 01:02:10.416436 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9846
I20260924 01:02:10.428253 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.417383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.8.254.65:35109)
0924 01:02:10.417535 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.417539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.417952 (+ 413us) server_negotiation.cc:407] Connection header received
0924 01:02:10.418071 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.418075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.418141 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.418272 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:10.419349 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.419991 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.421101 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.421494 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.421611 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.423725 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:10.423729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:10.423732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:10.423760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:10.426039 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.426674 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.426677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.426680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.426797 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.427099 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.427103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.427106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.427346 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:10.427573 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.427850 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.428003 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 01:02:10.428887 9969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:10.518257 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9846
I20260924 01:02:10.536288 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9980
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:10.552016 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.537011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.8.254.66:43271)
0924 01:02:10.537529 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.537534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.537548 (+ 14us) server_negotiation.cc:407] Connection header received
0924 01:02:10.537601 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.537604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.537678 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.537828 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:10.539099 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.539805 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.541162 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.541711 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.541850 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.545933 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:10.545949 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:10.545966 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:10.546001 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:10.549028 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.549748 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.549757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.549759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.549902 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.550459 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.550464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.550468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.550819 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:10.551131 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.551437 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.551657 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
I20260924 01:02:10.552670 10103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:10.643412 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9980
I20260924 01:02:10.660718 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10114
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[9740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:10.676528 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:10.661592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.8.254.67:36759)
0924 01:02:10.661777 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:10.661782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:10.661801 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:02:10.661943 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:10.661947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:10.662014 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:10.662102 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:10.663490 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.664276 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.665493 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:10.665859 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:10.665956 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:10.670643 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:10.670661 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:10.670677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:10.670733 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:10.673637 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.674439 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.674448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.674451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.674624 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:10.675007 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:10.675011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:10.675013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:10.675405 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:10.675663 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:10.675976 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 01:02:10.676187 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:02:10.677215 10237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:10.766541 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 10114
I20260924 01:02:10.783699 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 9753
I20260924 01:02:10.784974 9815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:10.893254 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 9753
2026-09-24T01:02:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4546 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[10292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[10292](info): set up 2 sockets
Sep 24 01:02:10 dist-test-slave-5c7l krb5kdc[10292](info): commencing operation
krb5kdc: starting...
W20260924 01:02:11.987351 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:12 dist-test-slave-5c7l krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:12.178169 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45723
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:12.507306 10305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:12.507598 10305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:12.507753 10305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:12.515347 10305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:12.515453 10305 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 01:02:12.515493 10305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:12.515612 10305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:12.515697 10305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:12.515810 10305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:12.526051 10305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:12.527439 10305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:12.528782 10305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:12.540874 10310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:12.540886 10313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:12.540911 10311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:12.541894 10305 server_base.cc:1034] running on GCE node
I20260924 01:02:12.542790 10305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:12.544227 10305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:12.545495 10305 hybrid_clock.cc:648] HybridClock initialized: now 1790211732545417 us; error 52 us; skew 500 ppm
I20260924 01:02:12.545845 10305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:12 dist-test-slave-5c7l krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211732, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:12.549095 10305 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:12.550668 10305 webserver.cc:460] Webserver started at http://127.8.254.126:39047/ using document root <none> and password file <none>
I20260924 01:02:12.551343 10305 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:12.551430 10305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:12.551708 10305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:12.554721 10305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "b6068efbbc6c468d91d8a33c5b48161a"
format_stamp: "Formatted at 2026-09-24 01:02:12 on dist-test-slave-5c7l"
server_key: "4f18cf5537cd43a8e94a977dfae3e8c2"
server_key_iv: "5b5b5755fe34a7d3981a80c41e3787a9"
server_key_version: "encryptionkey@0"
I20260924 01:02:12.555588 10305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "b6068efbbc6c468d91d8a33c5b48161a"
format_stamp: "Formatted at 2026-09-24 01:02:12 on dist-test-slave-5c7l"
server_key: "4f18cf5537cd43a8e94a977dfae3e8c2"
server_key_iv: "5b5b5755fe34a7d3981a80c41e3787a9"
server_key_version: "encryptionkey@0"
I20260924 01:02:12.560848 10305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:12.564602 10320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:12.566084 10305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:12.566310 10305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "b6068efbbc6c468d91d8a33c5b48161a"
format_stamp: "Formatted at 2026-09-24 01:02:12 on dist-test-slave-5c7l"
server_key: "4f18cf5537cd43a8e94a977dfae3e8c2"
server_key_iv: "5b5b5755fe34a7d3981a80c41e3787a9"
server_key_version: "encryptionkey@0"
I20260924 01:02:12.566525 10305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:12.584595 10305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:12.589042 10305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:12.589353 10305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:12.611591 10305 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45723
I20260924 01:02:12.611625 10381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45723 every 8 connection(s)
I20260924 01:02:12.613082 10305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:12.616549 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 10305
I20260924 01:02:12.616789 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:12.617012 10382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:12.617472 9209 external_mini_cluster.cc:1428] Setting key 6532e57f1de76982c360bd57d0c9c2e8
I20260924 01:02:12.631109 10382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: Bootstrap starting.
Sep 24 01:02:12 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:12.635102 10382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:12.636371 10382 log.cc:824] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:12.639451 10382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: No bootstrap required, opened a new log
I20260924 01:02:12.642033 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:12.619558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.8.254.126:45723)
0924 01:02:12.620451 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:12.620463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:12.620503 (+ 40us) server_negotiation.cc:407] Connection header received
0924 01:02:12.622956 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:12.622981 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:12.623434 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:12.623790 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:12.625584 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:12.626959 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:12.628783 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:12.629466 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:12.629600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:12.634495 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:12.634504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:12.634520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:12.634562 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:12.637762 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:12.638594 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:12.638601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:12.638608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:12.638786 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:12.639160 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:12.639165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:12.639168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:12.639808 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:12.640136 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:12.640549 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 01:02:12.640985 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":418,"threads_started":1}
I20260924 01:02:12.648167 10382 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6068efbbc6c468d91d8a33c5b48161a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45723 } }
I20260924 01:02:12.648648 10382 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:12.648752 10382 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6068efbbc6c468d91d8a33c5b48161a, State: Initialized, Role: FOLLOWER
I20260924 01:02:12.649269 10382 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6068efbbc6c468d91d8a33c5b48161a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45723 } }
I20260924 01:02:12.649470 10382 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:12.649564 10382 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:12.649689 10382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:12.652706 10382 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6068efbbc6c468d91d8a33c5b48161a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45723 } }
I20260924 01:02:12.653265 10382 leader_election.cc:302] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6068efbbc6c468d91d8a33c5b48161a; no voters:
I20260924 01:02:12.653828 10382 leader_election.cc:288] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:12.653981 10387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:12.654786 10387 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [term 1 LEADER]: Becoming Leader. State: Replica: b6068efbbc6c468d91d8a33c5b48161a, State: Running, Role: LEADER
I20260924 01:02:12.655360 10387 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6068efbbc6c468d91d8a33c5b48161a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45723 } }
I20260924 01:02:12.656111 10382 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:12.658180 10389 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6068efbbc6c468d91d8a33c5b48161a. Latest consensus state: current_term: 1 leader_uuid: "b6068efbbc6c468d91d8a33c5b48161a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6068efbbc6c468d91d8a33c5b48161a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45723 } } }
I20260924 01:02:12.658329 10388 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6068efbbc6c468d91d8a33c5b48161a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6068efbbc6c468d91d8a33c5b48161a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45723 } } }
I20260924 01:02:12.658587 10389 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:12.659319 10388 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:12.660450 10392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:12.666424 10392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:12.675597 10392 catalog_manager.cc:1347] Generated new cluster ID: e790cfce008a4a78a46b506234dd901e
I20260924 01:02:12.675737 10392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:12.707214 10392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:12.709111 10392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:12.720326 10392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: Generated new TSK 0
I20260924 01:02:12.721221 10392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:12.778880 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45723
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:13.106720 10408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:13.107066 10408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:13.107239 10408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:13.114606 10408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:13.114769 10408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:13.114923 10408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:02:13.115062 10408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:13.124842 10408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:45723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:13.126303 10408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:13.127676 10408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:13.142576 10413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:13.142611 10416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:13.142673 10414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:13.143356 10408 server_base.cc:1034] running on GCE node
I20260924 01:02:13.144043 10408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:13.145037 10408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:13.146332 10408 hybrid_clock.cc:648] HybridClock initialized: now 1790211733146266 us; error 38 us; skew 500 ppm
I20260924 01:02:13.146656 10408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:13 dist-test-slave-5c7l krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211733, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:13.164718 10408 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:13.166496 10408 webserver.cc:460] Webserver started at http://127.8.254.65:44601/ using document root <none> and password file <none>
I20260924 01:02:13.167356 10408 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:13.167500 10408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:13.167816 10408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:13.171010 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "392d78cfc2a8429c8f07cd539e1bd4b2"
format_stamp: "Formatted at 2026-09-24 01:02:13 on dist-test-slave-5c7l"
server_key: "d1915949489732870fbfd6429a89b426"
server_key_iv: "3cca81b108f73323e0a19def5b10d3a4"
server_key_version: "encryptionkey@0"
I20260924 01:02:13.171841 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "392d78cfc2a8429c8f07cd539e1bd4b2"
format_stamp: "Formatted at 2026-09-24 01:02:13 on dist-test-slave-5c7l"
server_key: "d1915949489732870fbfd6429a89b426"
server_key_iv: "3cca81b108f73323e0a19def5b10d3a4"
server_key_version: "encryptionkey@0"
I20260924 01:02:13.177242 10408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:13.180827 10423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:13.182265 10408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:13.182444 10408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "392d78cfc2a8429c8f07cd539e1bd4b2"
format_stamp: "Formatted at 2026-09-24 01:02:13 on dist-test-slave-5c7l"
server_key: "d1915949489732870fbfd6429a89b426"
server_key_iv: "3cca81b108f73323e0a19def5b10d3a4"
server_key_version: "encryptionkey@0"
I20260924 01:02:13.182708 10408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:13.204859 10408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:13.209558 10408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:13.209862 10408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:13.212702 10408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:13.212815 10408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:13.212975 10408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:13.213084 10408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:02:13 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211733, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:13.237659 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.213339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:56831 (local address 127.8.254.126:45723)
0924 01:02:13.214598 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:13.214604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:13.215468 (+ 864us) server_negotiation.cc:407] Connection header received
0924 01:02:13.217644 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:13.217651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:13.217751 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:13.217843 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:13.220401 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.221258 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.223900 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.224703 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.224860 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.229565 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:13.229576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:13.229584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:13.229641 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:13.233808 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.234800 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.234807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.234810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.234993 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.235600 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.235628 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.235633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.236051 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:13.236206 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:13.237041 (+ 835us) server_negotiation.cc:299] Negotiation successful
0924 01:02:13.237289 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":1002,"threads_started":1}
I20260924 01:02:13.239037 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.214273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.8.254.65:56831)
0924 01:02:13.215274 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:13.215325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:13.217271 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:13.218439 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:13.218452 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:13.219343 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:13.220186 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.220220 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.221416 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.221423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.223715 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.223739 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.224964 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.225009 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.225373 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.226312 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:13.226357 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:13.229257 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:13.234002 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.234013 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.234031 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.234616 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.235162 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.235168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.235174 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.235461 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.236206 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:13.236216 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:13.236646 (+ 430us) client_negotiation.cc:769] Sending connection context
0924 01:02:13.236933 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 01:02:13.237498 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":424,"threads_started":1}
I20260924 01:02:13.264334 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.246533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.8.254.65:35607)
0924 01:02:13.247445 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:13.247472 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:13.247656 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:13.248014 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:13.248021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:13.248773 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:13.249247 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.249265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.250316 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.250325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.252611 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.252637 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.254127 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.254137 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.254451 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.255302 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:13.255335 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:13.257036 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:13.261680 (+ 4644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.261686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.261689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.262125 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.262623 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.262626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.262628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.262750 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.263485 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:13.263488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:13.263642 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 01:02:13.263749 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 01:02:13.263998 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":494,"threads_started":1}
I20260924 01:02:13.264760 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.246490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:35607 (local address 127.8.254.126:45723)
0924 01:02:13.246694 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:13.246701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:13.247572 (+ 871us) server_negotiation.cc:407] Connection header received
0924 01:02:13.247783 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:13.247786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:13.247851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:13.247933 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:13.249411 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.250119 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.252857 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.253636 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.253755 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.257359 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:13.257369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:13.257374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:13.257422 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:13.261527 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.262282 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.262290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.262297 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.262490 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.262877 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.262885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.262890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.263362 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:13.263544 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:13.263919 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 01:02:13.264248 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 01:02:13.274158 10408 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:35933
I20260924 01:02:13.274179 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:35933 every 8 connection(s)
I20260924 01:02:13.275525 10408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:13.277640 10544 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45723
I20260924 01:02:13.277830 10544 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:13.278489 10544 heartbeater.cc:511] Master 127.8.254.126:45723 requested a full tablet report, sending...
I20260924 01:02:13.279356 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 10408
I20260924 01:02:13.279584 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:13.280251 9209 external_mini_cluster.cc:1428] Setting key fbbb736362bd18ad2595fc68b0a39e0c
I20260924 01:02:13.280663 10337 ts_manager.cc:194] Registered new tserver with Master: 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933)
I20260924 01:02:13.282454 10337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:56831
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:13.321941 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45723
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:13.658851 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:13.659204 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:13.659376 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:13.666867 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:13.667035 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:13.667186 10548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:02:13.667335 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:13.677889 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:45723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:13.679378 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:13.680928 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:13.695281 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:13.695307 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:13.695353 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:13.696501 10548 server_base.cc:1034] running on GCE node
I20260924 01:02:13.697249 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:13.698313 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:13.699568 10548 hybrid_clock.cc:648] HybridClock initialized: now 1790211733699503 us; error 42 us; skew 500 ppm
I20260924 01:02:13.699924 10548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:13 dist-test-slave-5c7l krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211733, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:13.703100 10548 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:13.704631 10548 webserver.cc:460] Webserver started at http://127.8.254.66:45171/ using document root <none> and password file <none>
I20260924 01:02:13.705345 10548 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:13.705477 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:13.705808 10548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:13.708989 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "4b249787f0b8411bb1b8aaef68e51c80"
format_stamp: "Formatted at 2026-09-24 01:02:13 on dist-test-slave-5c7l"
server_key: "cacc564b25c7f7665ff40e425b4a6071"
server_key_iv: "2bcde1217d0a8b540e516d850fa0c930"
server_key_version: "encryptionkey@0"
I20260924 01:02:13.709826 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "4b249787f0b8411bb1b8aaef68e51c80"
format_stamp: "Formatted at 2026-09-24 01:02:13 on dist-test-slave-5c7l"
server_key: "cacc564b25c7f7665ff40e425b4a6071"
server_key_iv: "2bcde1217d0a8b540e516d850fa0c930"
server_key_version: "encryptionkey@0"
I20260924 01:02:13.715348 10548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:02:13.718847 10564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:13.720181 10548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:13.720358 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "4b249787f0b8411bb1b8aaef68e51c80"
format_stamp: "Formatted at 2026-09-24 01:02:13 on dist-test-slave-5c7l"
server_key: "cacc564b25c7f7665ff40e425b4a6071"
server_key_iv: "2bcde1217d0a8b540e516d850fa0c930"
server_key_version: "encryptionkey@0"
I20260924 01:02:13.720592 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:13.734984 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:13.739377 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:13.739681 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:13.742758 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:13.742868 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:13.743034 10548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:13.743119 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:02:13 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211733, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:13.769452 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.743283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:46299 (local address 127.8.254.126:45723)
0924 01:02:13.743493 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:13.743502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:13.744971 (+ 1469us) server_negotiation.cc:407] Connection header received
0924 01:02:13.747116 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:13.747124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:13.747217 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:13.747332 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:13.749737 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.750868 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.754358 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.755022 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.755131 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.761453 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:13.761459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:13.761462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:13.761530 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:13.765517 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.766462 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.766471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.766477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.766677 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.767156 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.767164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.767170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.767649 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:13.767850 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:13.768593 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 01:02:13.768909 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:02:13.770293 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.743974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.8.254.66:46299)
0924 01:02:13.744802 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:13.744842 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:13.746822 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:13.747592 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:13.747601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:13.748451 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:13.749451 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.749483 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.751102 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.751111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.754056 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.754090 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.755314 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.755324 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.755710 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.756725 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:13.756764 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:13.761051 (+ 4287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:13.765736 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.765747 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.765772 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.766302 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.766807 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.766811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.766813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.767027 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.767794 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:13.767800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:13.768176 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 01:02:13.768416 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 01:02:13.768849 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260924 01:02:13.794168 10548 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:34677
I20260924 01:02:13.794170 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:34677 every 8 connection(s)
I20260924 01:02:13.795876 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:13.796859 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.777387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:55489 (local address 127.8.254.126:45723)
0924 01:02:13.777639 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:13.777657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:13.778698 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 01:02:13.779047 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:13.779055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:13.779153 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:13.779242 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:13.781100 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.782385 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.785222 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.785669 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.785785 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.789170 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:13.789179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:13.789186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:13.789252 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:13.793415 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.794613 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.794620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.794629 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.794783 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:13.795205 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:13.795210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:13.795213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:13.795550 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:13.795666 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:13.796244 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 01:02:13.796481 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:02:13.797178 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:13.777790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.8.254.66:55489)
0924 01:02:13.778539 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:13.778563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:13.778849 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:13.779515 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:13.779521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:13.780296 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:13.780865 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.780887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.782573 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.782583 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.784990 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:13.785010 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:13.785891 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:13.785901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:13.786161 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:13.787252 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:13.787281 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:13.788903 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:13.793679 (+ 4776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.793684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.793687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.794418 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.794918 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:13.794922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:13.794926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:13.795082 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:13.795677 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:13.795682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:13.795925 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 01:02:13.796096 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 01:02:13.796508 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":444,"threads_started":1}
I20260924 01:02:13.798584 10685 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45723
I20260924 01:02:13.798843 10685 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:13.799541 10685 heartbeater.cc:511] Master 127.8.254.126:45723 requested a full tablet report, sending...
I20260924 01:02:13.800889 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 10548
I20260924 01:02:13.801095 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:13.801072 10337 ts_manager.cc:194] Registered new tserver with Master: 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677)
I20260924 01:02:13.801625 9209 external_mini_cluster.cc:1428] Setting key e0e67c610feddd4c75de246871604a5b
I20260924 01:02:13.802121 10337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:46299
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:13.842842 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45723
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:14.172214 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:14.172523 10689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:14.172706 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:14.180198 10689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:14.180317 10689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:14.180481 10689 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:02:14.180612 10689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:14.191025 10689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:45723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:14.192411 10689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:14.193837 10689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:14.209395 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:14.209367 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:14.209439 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:14.210301 10689 server_base.cc:1034] running on GCE node
I20260924 01:02:14.210908 10689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:14.211840 10689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:14.213060 10689 hybrid_clock.cc:648] HybridClock initialized: now 1790211734213000 us; error 34 us; skew 500 ppm
I20260924 01:02:14.213357 10689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:14 dist-test-slave-5c7l krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:14.216635 10689 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:14.218057 10689 webserver.cc:460] Webserver started at http://127.8.254.67:43693/ using document root <none> and password file <none>
I20260924 01:02:14.218799 10689 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:14.218927 10689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:14.219273 10689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:14.222106 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "b0452ebaca064d2ab2e184f968ca6d9b"
format_stamp: "Formatted at 2026-09-24 01:02:14 on dist-test-slave-5c7l"
server_key: "b2f555e72c207ee4553174a69ea35d6a"
server_key_iv: "edff01deb897cb50e7e03abec4f8e328"
server_key_version: "encryptionkey@0"
I20260924 01:02:14.222986 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "b0452ebaca064d2ab2e184f968ca6d9b"
format_stamp: "Formatted at 2026-09-24 01:02:14 on dist-test-slave-5c7l"
server_key: "b2f555e72c207ee4553174a69ea35d6a"
server_key_iv: "edff01deb897cb50e7e03abec4f8e328"
server_key_version: "encryptionkey@0"
I20260924 01:02:14.228394 10689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 01:02:14.231995 10704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:14.233309 10689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:02:14.233489 10689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "b0452ebaca064d2ab2e184f968ca6d9b"
format_stamp: "Formatted at 2026-09-24 01:02:14 on dist-test-slave-5c7l"
server_key: "b2f555e72c207ee4553174a69ea35d6a"
server_key_iv: "edff01deb897cb50e7e03abec4f8e328"
server_key_version: "encryptionkey@0"
I20260924 01:02:14.233713 10689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:14.258911 10689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:14.263501 10689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:14.263788 10689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:14.266539 10689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:14.266649 10689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:14.266815 10689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:14.267136 10689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:02:14 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:14.286046 10544 heartbeater.cc:503] Master 127.8.254.126:45723 was elected leader, sending a full tablet report...
I20260924 01:02:14.294144 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.267064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:45945 (local address 127.8.254.126:45723)
0924 01:02:14.267265 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.267271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.268887 (+ 1616us) server_negotiation.cc:407] Connection header received
0924 01:02:14.271231 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.271237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.271334 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.271444 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:14.273993 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.275392 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.278793 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.279600 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.279749 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.285058 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:14.285065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:14.285068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:14.285104 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:14.288531 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:14.289452 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:14.289457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:14.289459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:14.289660 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:14.290390 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:14.290399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:14.290402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:14.290823 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:14.290995 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.293551 (+ 2556us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.293808 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:02:14.294631 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.267806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.8.254.67:45945)
0924 01:02:14.268688 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.268739 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.270910 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.271725 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.271740 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.272630 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:14.273630 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.273750 (+ 120us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.275639 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.275646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.278555 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.278577 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.280035 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.280044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.280301 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.281107 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:14.281153 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:14.284797 (+ 3644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:14.288691 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:14.288701 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:14.288720 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:14.289311 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:14.289805 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:14.289813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:14.289820 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:14.290170 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:14.291078 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:14.291090 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:14.291733 (+ 643us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.292148 (+ 415us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.292935 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":378,"threads_started":1}
I20260924 01:02:14.316649 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.301370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.8.254.67:46763)
0924 01:02:14.302277 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.302300 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.302472 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.302896 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.302901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.303372 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:14.303653 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.303663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.304962 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.304967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.305991 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.306002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.307018 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.307030 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.307301 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.308183 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:14.308214 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:14.309310 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:14.313003 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:14.313009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:14.313014 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:14.313659 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:14.314089 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:14.314094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:14.314098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:14.314338 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:14.315057 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:14.315066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:14.315380 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.315582 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.316051 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":568,"threads_started":1}
I20260924 01:02:14.317633 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.301256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:46763 (local address 127.8.254.126:45723)
0924 01:02:14.301466 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.301473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.302399 (+ 926us) server_negotiation.cc:407] Connection header received
0924 01:02:14.302679 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.302684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.302760 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.302864 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:14.303800 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.304813 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.306141 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.306765 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.306939 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.309522 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:14.309527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:14.309530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:14.309562 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:14.312856 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:14.313777 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:14.313784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:14.313787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:14.313942 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:14.314480 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:14.314488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:14.314493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:14.314903 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:14.315087 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.316966 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.317218 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:02:14.318444 10689 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:39053
I20260924 01:02:14.318524 10824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:39053 every 8 connection(s)
I20260924 01:02:14.319806 10689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:14.321457 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 10689
I20260924 01:02:14.321668 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:14.322252 9209 external_mini_cluster.cc:1428] Setting key 98df7fcd060a54ce7f1b5e8cb4897740
I20260924 01:02:14.322412 10825 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45723
I20260924 01:02:14.322656 10825 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:14.323457 10825 heartbeater.cc:511] Master 127.8.254.126:45723 requested a full tablet report, sending...
I20260924 01:02:14.324849 10336 ts_manager.cc:194] Registered new tserver with Master: b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053)
I20260924 01:02:14.325863 10336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:45945
I20260924 01:02:14.338457 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 01:02:14 dist-test-slave-5c7l krb5kdc[10292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 01:02:14 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:14.385807 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.371899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.8.254.126:45723)
0924 01:02:14.372149 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.372155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.372511 (+ 356us) server_negotiation.cc:407] Connection header received
0924 01:02:14.372695 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.372699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.372782 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.372870 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:14.374453 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.375191 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.376422 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.376905 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.376996 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.380871 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:14.380876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:14.380879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:14.380913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:14.383465 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:14.384083 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:14.384086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:14.384089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:14.384209 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:14.384599 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:14.384602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:14.384604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:14.384863 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:14.385045 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.385351 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.385517 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 01:02:14.390751 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:14.398522 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:14.422762 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.411307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.8.254.67:39053)
0924 01:02:14.413680 (+ 2373us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.413693 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.413734 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:02:14.413858 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.413870 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.414221 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.415649 (+ 1428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.416005 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.417609 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.420076 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.421571 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.421714 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.421973 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.422223 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.422389 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2160,"thread_start_us":1934,"threads_started":1}
I20260924 01:02:14.422737 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.410987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39053 (local address 127.0.0.1:49090)
0924 01:02:14.412728 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.412752 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.412899 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.414519 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.414530 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.414584 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.415400 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.415419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.417744 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.417753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.419894 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.419911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.421761 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.421767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.421952 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.422006 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.422158 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.422393 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":2465,"threads_started":1}
I20260924 01:02:14.424014 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.412942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:34677 (local address 127.0.0.1:41892)
0924 01:02:14.415102 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.415125 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.415281 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.416385 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.416391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.416415 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.416813 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.416829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.418760 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.418766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.421053 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.421069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.423131 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.423140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.423362 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.423392 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.423479 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.423612 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2013,"mutex_wait_us":4,"thread_start_us":1862,"threads_started":1}
I20260924 01:02:14.424104 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.410937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.8.254.65:35933)
0924 01:02:14.416126 (+ 5189us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.416135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.416160 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:02:14.416250 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.416258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.416482 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.417436 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.417631 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.419040 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.421497 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.422929 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.423356 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.423572 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.423653 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.423815 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5044,"thread_start_us":4924,"threads_started":1}
I20260924 01:02:14.424116 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.410821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35933 (local address 127.0.0.1:47220)
0924 01:02:14.411683 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.411740 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.411949 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.416704 (+ 4755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.416708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.416749 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.417244 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.417258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.419184 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.419190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.421331 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.421346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.423186 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.423192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.423450 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.423486 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.423623 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.423754 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":49,"thread_start_us":662,"threads_started":1}
I20260924 01:02:14.424726 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.413472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.8.254.66:34677)
0924 01:02:14.415781 (+ 2309us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.415791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.415822 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:02:14.415931 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.415939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.416168 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.416385 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.417149 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.418622 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.421255 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.422618 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.422752 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.422944 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.424279 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.424450 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2164,"thread_start_us":2041,"threads_started":1}
I20260924 01:02:14.427529 10754 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec6150f65254c379bfb92c774d9e546 (DEFAULT_TABLE table=test-table [id=0818746001a94b34820fa94b709ee2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:14.427966 10474 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec6150f65254c379bfb92c774d9e546 (DEFAULT_TABLE table=test-table [id=0818746001a94b34820fa94b709ee2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:14.428113 10614 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec6150f65254c379bfb92c774d9e546 (DEFAULT_TABLE table=test-table [id=0818746001a94b34820fa94b709ee2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:14.429174 10474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec6150f65254c379bfb92c774d9e546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:14.429191 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec6150f65254c379bfb92c774d9e546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:14.429382 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec6150f65254c379bfb92c774d9e546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:14.440819 10844 tablet_bootstrap.cc:492] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Bootstrap starting.
I20260924 01:02:14.445225 10844 tablet_bootstrap.cc:654] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:14.445264 10846 tablet_bootstrap.cc:492] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Bootstrap starting.
I20260924 01:02:14.446856 10844 log.cc:824] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:14.447134 10845 tablet_bootstrap.cc:492] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Bootstrap starting.
I20260924 01:02:14.449383 10846 tablet_bootstrap.cc:654] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:14.450223 10845 tablet_bootstrap.cc:654] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:14.450760 10846 log.cc:824] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:14.451324 10845 log.cc:824] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:14.452893 10844 tablet_bootstrap.cc:492] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: No bootstrap required, opened a new log
I20260924 01:02:14.453234 10844 ts_tablet_manager.cc:1397] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260924 01:02:14.454231 10846 tablet_bootstrap.cc:492] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: No bootstrap required, opened a new log
I20260924 01:02:14.454582 10846 ts_tablet_manager.cc:1397] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 01:02:14.460518 10845 tablet_bootstrap.cc:492] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: No bootstrap required, opened a new log
I20260924 01:02:14.460875 10845 ts_tablet_manager.cc:1397] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260924 01:02:14.461218 10844 raft_consensus.cc:348] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.461628 10844 raft_consensus.cc:374] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:14.461781 10844 raft_consensus.cc:729] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392d78cfc2a8429c8f07cd539e1bd4b2, State: Initialized, Role: FOLLOWER
I20260924 01:02:14.462502 10844 consensus_queue.cc:260] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.465711 10846 raft_consensus.cc:348] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.466163 10846 raft_consensus.cc:374] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:14.466343 10846 raft_consensus.cc:729] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0452ebaca064d2ab2e184f968ca6d9b, State: Initialized, Role: FOLLOWER
I20260924 01:02:14.467027 10846 consensus_queue.cc:260] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.468485 10844 ts_tablet_manager.cc:1428] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260924 01:02:14.469724 10825 heartbeater.cc:503] Master 127.8.254.126:45723 was elected leader, sending a full tablet report...
I20260924 01:02:14.470443 10846 ts_tablet_manager.cc:1428] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260924 01:02:14.472645 10845 raft_consensus.cc:348] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.473097 10845 raft_consensus.cc:374] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:14.473242 10845 raft_consensus.cc:729] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b249787f0b8411bb1b8aaef68e51c80, State: Initialized, Role: FOLLOWER
I20260924 01:02:14.473925 10845 consensus_queue.cc:260] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.474959 10685 heartbeater.cc:503] Master 127.8.254.126:45723 was elected leader, sending a full tablet report...
I20260924 01:02:14.475409 10845 ts_tablet_manager.cc:1428] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260924 01:02:14.527923 10545 tablet.cc:2369] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:14.549377 10686 tablet.cc:2369] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:14.572108 10826 tablet.cc:2369] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:02:14.868221 10850 raft_consensus.cc:482] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:14.868599 10850 raft_consensus.cc:504] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.869344 10852 raft_consensus.cc:482] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:14.869748 10852 raft_consensus.cc:504] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.870625 10850 leader_election.cc:288] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053), 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677)
I20260924 01:02:14.875658 10852 leader_election.cc:288] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933), b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053)
I20260924 01:02:14.880308 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.870994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39053 (local address 127.8.254.65:58393)
0924 01:02:14.871910 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.871926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.872088 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.873189 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.873195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.873222 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.873627 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.873641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.875081 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.875086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.877553 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.877565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.879414 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.879420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.879591 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.879632 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.879724 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.879876 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":1,"thread_start_us":586,"threads_started":1}
I20260924 01:02:14.881036 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.871032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:58393 (local address 127.8.254.67:39053)
0924 01:02:14.871228 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.871236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.872017 (+ 781us) server_negotiation.cc:407] Connection header received
0924 01:02:14.872246 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.872253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.872348 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.872444 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.873825 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.874930 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.877754 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.879022 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.879191 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.879271 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.880540 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.880772 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":53}
I20260924 01:02:14.881244 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.871398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:41753 (local address 127.8.254.66:34677)
0924 01:02:14.871645 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.871652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.872885 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 01:02:14.873061 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.873069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.873152 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.873214 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.874109 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.875632 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.878484 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.879838 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.879984 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.880082 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.880830 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.880963 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":73}
I20260924 01:02:14.882020 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.871173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:34677 (local address 127.8.254.65:41753)
0924 01:02:14.872691 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.872710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.872875 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.873432 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.873436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.873462 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.873933 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.873944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.875872 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.875881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.878290 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.878305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.880280 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.880288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.880638 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.880676 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.881469 (+ 793us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.881606 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":61,"thread_start_us":1163,"threads_started":1}
I20260924 01:02:14.882078 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" is_pre_election: true
I20260924 01:02:14.882616 10774 raft_consensus.cc:2454] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 392d78cfc2a8429c8f07cd539e1bd4b2 in term 0.
I20260924 01:02:14.882936 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.873130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54451 (local address 127.8.254.67:39053)
0924 01:02:14.874664 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.874672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.874695 (+ 23us) server_negotiation.cc:407] Connection header received
0924 01:02:14.874750 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.874754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.874813 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.874909 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.876563 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.877621 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.880304 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.881562 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.881705 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.881774 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.882376 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.882525 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1433,"thread_start_us":1271,"threads_started":1}
I20260924 01:02:14.882838 10638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b249787f0b8411bb1b8aaef68e51c80" is_pre_election: true
I20260924 01:02:14.883276 10638 raft_consensus.cc:2454] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 392d78cfc2a8429c8f07cd539e1bd4b2 in term 0.
I20260924 01:02:14.883697 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.872675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39053 (local address 127.8.254.66:54451)
0924 01:02:14.873687 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.873762 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.873905 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.875080 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.875085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.875115 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.876377 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.876391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.877811 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.877818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.880128 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.880142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.881977 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.881985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.882164 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.882232 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.882662 (+ 430us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.883280 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":760,"thread_start_us":3410,"threads_started":1}
I20260924 01:02:14.883821 10424 leader_election.cc:302] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392d78cfc2a8429c8f07cd539e1bd4b2, b0452ebaca064d2ab2e184f968ca6d9b; no voters:
I20260924 01:02:14.884248 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" is_pre_election: true
I20260924 01:02:14.884424 10774 raft_consensus.cc:2454] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b249787f0b8411bb1b8aaef68e51c80 in term 0.
I20260924 01:02:14.884536 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.873026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:50005 (local address 127.8.254.65:35933)
0924 01:02:14.873198 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.873204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.876468 (+ 3264us) server_negotiation.cc:407] Connection header received
0924 01:02:14.876688 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.876693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.876776 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.876833 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:14.877553 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.878741 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.882071 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.883283 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.883401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.883472 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.884177 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.884306 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260924 01:02:14.884737 10850 raft_consensus.cc:2781] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:14.884850 10565 leader_election.cc:302] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b249787f0b8411bb1b8aaef68e51c80, b0452ebaca064d2ab2e184f968ca6d9b; no voters:
I20260924 01:02:14.884899 10850 raft_consensus.cc:482] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:14.885016 10850 raft_consensus.cc:3037] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:14.885175 10852 raft_consensus.cc:2781] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:14.885326 10852 raft_consensus.cc:482] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:14.885425 10852 raft_consensus.cc:3037] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:14.885820 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.872930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35933 (local address 127.8.254.66:50005)
0924 01:02:14.876381 (+ 3451us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:14.876398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:14.876559 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:14.877007 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:14.877012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:14.877034 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:14.877364 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.877376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.878932 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.878939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.881871 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:14.881886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.883810 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.883818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:14.883991 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.884022 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 01:02:14.885239 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0924 01:02:14.885393 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3321,"mutex_wait_us":3,"thread_start_us":1157,"threads_started":1}
I20260924 01:02:14.887035 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" is_pre_election: true
I20260924 01:02:14.888077 10850 raft_consensus.cc:504] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.888523 10499 raft_consensus.cc:2379] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4b249787f0b8411bb1b8aaef68e51c80 in current term 1: Already voted for candidate 392d78cfc2a8429c8f07cd539e1bd4b2 in this term.
I20260924 01:02:14.888412 10852 raft_consensus.cc:504] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.889096 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452ebaca064d2ab2e184f968ca6d9b"
I20260924 01:02:14.889321 10774 raft_consensus.cc:3037] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:14.889503 10638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b249787f0b8411bb1b8aaef68e51c80"
I20260924 01:02:14.889660 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2"
I20260924 01:02:14.889938 10638 raft_consensus.cc:2379] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 392d78cfc2a8429c8f07cd539e1bd4b2 in current term 1: Already voted for candidate 4b249787f0b8411bb1b8aaef68e51c80 in this term.
I20260924 01:02:14.889990 10498 raft_consensus.cc:2379] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b249787f0b8411bb1b8aaef68e51c80 in current term 1: Already voted for candidate 392d78cfc2a8429c8f07cd539e1bd4b2 in this term.
I20260924 01:02:14.890488 10852 leader_election.cc:288] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 election: Requested vote from peers 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933), b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053)
I20260924 01:02:14.891144 10850 leader_election.cc:288] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [CANDIDATE]: Term 1 election: Requested vote from peers b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053), 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677)
I20260924 01:02:14.891224 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec6150f65254c379bfb92c774d9e546" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452ebaca064d2ab2e184f968ca6d9b"
I20260924 01:02:14.892042 10565 leader_election.cc:302] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b249787f0b8411bb1b8aaef68e51c80; no voters: 392d78cfc2a8429c8f07cd539e1bd4b2, b0452ebaca064d2ab2e184f968ca6d9b
I20260924 01:02:14.892306 10852 raft_consensus.cc:2726] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 01:02:14.892356 10774 raft_consensus.cc:2454] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 392d78cfc2a8429c8f07cd539e1bd4b2 in term 1.
I20260924 01:02:14.892707 10424 leader_election.cc:302] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 392d78cfc2a8429c8f07cd539e1bd4b2, b0452ebaca064d2ab2e184f968ca6d9b; no voters: 4b249787f0b8411bb1b8aaef68e51c80
I20260924 01:02:14.893006 10850 raft_consensus.cc:2781] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:14.893401 10850 raft_consensus.cc:686] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 LEADER]: Becoming Leader. State: Replica: 392d78cfc2a8429c8f07cd539e1bd4b2, State: Running, Role: LEADER
I20260924 01:02:14.893854 10850 consensus_queue.cc:237] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } }
I20260924 01:02:14.898182 10336 catalog_manager.cc:5495] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65). New cstate: current_term: 1 leader_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } health_report { overall_health: UNKNOWN } } }
I20260924 01:02:14.995844 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:14.989910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.8.254.65:35933)
0924 01:02:14.990081 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:14.990085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:14.990725 (+ 640us) server_negotiation.cc:407] Connection header received
0924 01:02:14.990998 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:14.991002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:14.991097 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:14.991232 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:14.991696 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.992373 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.993736 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:14.994108 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:14.994272 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:14.994717 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:14.995279 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:14.995386 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:14.995506 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 01:02:14.995575 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:02:15.004055 10774 raft_consensus.cc:1261] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Refusing update from remote peer 392d78cfc2a8429c8f07cd539e1bd4b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:15.004081 10638 raft_consensus.cc:1261] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Refusing update from remote peer 392d78cfc2a8429c8f07cd539e1bd4b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:15.004938 10858 consensus_queue.cc:1035] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:15.005388 10850 consensus_queue.cc:1035] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:15.017169 10862 mvcc.cc:205] Tried to move back new op lower bound from 7332707266566180864 to 7332707266128969728. Current Snapshot: MvccSnapshot[applied={T|T < 7332707266566180864}]
I20260924 01:02:15.023931 10864 mvcc.cc:205] Tried to move back new op lower bound from 7332707266566180864 to 7332707266128969728. Current Snapshot: MvccSnapshot[applied={T|T < 7332707266566180864}]
I20260924 01:02:15.037691 10336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:15.038007 10336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:15.041896 10336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: Sending DeleteTablet for 3 replicas of tablet 6ec6150f65254c379bfb92c774d9e546
I20260924 01:02:15.043218 10754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec6150f65254c379bfb92c774d9e546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:15 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49090
I20260924 01:02:15.043242 10614 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec6150f65254c379bfb92c774d9e546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:15 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:41892
I20260924 01:02:15.044901 10474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec6150f65254c379bfb92c774d9e546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:15 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:47220
I20260924 01:02:15.046797 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:15.047667 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:15.059175 10473 tablet_service.cc:1467] Processing CreateTablet for tablet 51fad49a993547a5b59974d85455f0d7 (DEFAULT_TABLE table=test-table [id=2fc8cd3a1b9d4f7aab51d56f26cfa4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:15.059336 10613 tablet_service.cc:1467] Processing CreateTablet for tablet 51fad49a993547a5b59974d85455f0d7 (DEFAULT_TABLE table=test-table [id=2fc8cd3a1b9d4f7aab51d56f26cfa4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:15.059810 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fad49a993547a5b59974d85455f0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:15.059875 10613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fad49a993547a5b59974d85455f0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:15.066938 10753 tablet_service.cc:1467] Processing CreateTablet for tablet 51fad49a993547a5b59974d85455f0d7 (DEFAULT_TABLE table=test-table [id=2fc8cd3a1b9d4f7aab51d56f26cfa4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:15.067559 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fad49a993547a5b59974d85455f0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:15.067879 10874 tablet_replica.cc:330] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: stopping tablet replica
I20260924 01:02:15.068166 10875 tablet_replica.cc:330] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: stopping tablet replica
I20260924 01:02:15.068193 10873 tablet_replica.cc:330] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: stopping tablet replica
I20260924 01:02:15.068671 10873 raft_consensus.cc:2229] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:15.068964 10874 raft_consensus.cc:2229] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:15.069314 10875 raft_consensus.cc:2229] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:15.070061 10874 raft_consensus.cc:2258] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:15.070365 10873 raft_consensus.cc:2258] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:15.070798 10875 raft_consensus.cc:2258] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:15.073922 10873 ts_tablet_manager.cc:1905] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:15.074299 10874 ts_tablet_manager.cc:1905] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:15.074958 10875 ts_tablet_manager.cc:1905] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:15.083292 10873 ts_tablet_manager.cc:1918] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:15.083483 10873 log.cc:1196] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/6ec6150f65254c379bfb92c774d9e546
I20260924 01:02:15.071908 10876 tablet_bootstrap.cc:492] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Bootstrap starting.
I20260924 01:02:15.084061 10873 ts_tablet_manager.cc:1939] T 6ec6150f65254c379bfb92c774d9e546 P b0452ebaca064d2ab2e184f968ca6d9b: Deleting consensus metadata
I20260924 01:02:15.085928 10876 tablet_bootstrap.cc:654] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:15.086148 10321 catalog_manager.cc:4841] TS b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053): tablet 6ec6150f65254c379bfb92c774d9e546 (table test-table [id=0818746001a94b34820fa94b709ee2a2]) successfully deleted
I20260924 01:02:15.086473 10877 tablet_bootstrap.cc:492] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Bootstrap starting.
I20260924 01:02:15.088688 10877 tablet_bootstrap.cc:654] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:15.089926 10874 ts_tablet_manager.cc:1918] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:15.090145 10874 log.cc:1196] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/6ec6150f65254c379bfb92c774d9e546
I20260924 01:02:15.090760 10874 ts_tablet_manager.cc:1939] T 6ec6150f65254c379bfb92c774d9e546 P 4b249787f0b8411bb1b8aaef68e51c80: Deleting consensus metadata
I20260924 01:02:15.092228 10875 ts_tablet_manager.cc:1918] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:15.092404 10875 log.cc:1196] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/6ec6150f65254c379bfb92c774d9e546
I20260924 01:02:15.092767 10323 catalog_manager.cc:4841] TS 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677): tablet 6ec6150f65254c379bfb92c774d9e546 (table test-table [id=0818746001a94b34820fa94b709ee2a2]) successfully deleted
I20260924 01:02:15.092967 10875 ts_tablet_manager.cc:1939] T 6ec6150f65254c379bfb92c774d9e546 P 392d78cfc2a8429c8f07cd539e1bd4b2: Deleting consensus metadata
I20260924 01:02:15.094378 10323 catalog_manager.cc:4841] TS 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933): tablet 6ec6150f65254c379bfb92c774d9e546 (table test-table [id=0818746001a94b34820fa94b709ee2a2]) successfully deleted
I20260924 01:02:15.094766 10878 tablet_bootstrap.cc:492] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Bootstrap starting.
I20260924 01:02:15.096752 10878 tablet_bootstrap.cc:654] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:15.097967 10877 tablet_bootstrap.cc:492] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: No bootstrap required, opened a new log
I20260924 01:02:15.098304 10877 ts_tablet_manager.cc:1397] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260924 01:02:15.098434 10876 tablet_bootstrap.cc:492] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: No bootstrap required, opened a new log
I20260924 01:02:15.098593 10876 ts_tablet_manager.cc:1397] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20260924 01:02:15.099460 10876 raft_consensus.cc:348] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.099742 10876 raft_consensus.cc:374] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:15.099848 10876 raft_consensus.cc:729] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392d78cfc2a8429c8f07cd539e1bd4b2, State: Initialized, Role: FOLLOWER
I20260924 01:02:15.099589 10877 raft_consensus.cc:348] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.100193 10877 raft_consensus.cc:374] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:15.100139 10876 consensus_queue.cc:260] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.100378 10877 raft_consensus.cc:729] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b249787f0b8411bb1b8aaef68e51c80, State: Initialized, Role: FOLLOWER
I20260924 01:02:15.100880 10877 consensus_queue.cc:260] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.103346 10876 ts_tablet_manager.cc:1428] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 01:02:15.105676 10877 ts_tablet_manager.cc:1428] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260924 01:02:15.105726 10878 tablet_bootstrap.cc:492] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: No bootstrap required, opened a new log
I20260924 01:02:15.105885 10878 ts_tablet_manager.cc:1397] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260924 01:02:15.106710 10878 raft_consensus.cc:348] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.106952 10878 raft_consensus.cc:374] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:15.107059 10878 raft_consensus.cc:729] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0452ebaca064d2ab2e184f968ca6d9b, State: Initialized, Role: FOLLOWER
I20260924 01:02:15.107292 10878 consensus_queue.cc:260] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.107906 10878 ts_tablet_manager.cc:1428] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 01:02:15.171351 10852 raft_consensus.cc:482] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:15.171591 10852 raft_consensus.cc:504] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.172340 10852 leader_election.cc:288] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933), b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053)
I20260924 01:02:15.172938 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fad49a993547a5b59974d85455f0d7" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" is_pre_election: true
I20260924 01:02:15.173082 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fad49a993547a5b59974d85455f0d7" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" is_pre_election: true
I20260924 01:02:15.173257 10499 raft_consensus.cc:2454] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b249787f0b8411bb1b8aaef68e51c80 in term 0.
I20260924 01:02:15.173389 10774 raft_consensus.cc:2454] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b249787f0b8411bb1b8aaef68e51c80 in term 0.
I20260924 01:02:15.173852 10567 leader_election.cc:302] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392d78cfc2a8429c8f07cd539e1bd4b2, 4b249787f0b8411bb1b8aaef68e51c80; no voters:
I20260924 01:02:15.174271 10852 raft_consensus.cc:2781] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:15.174382 10852 raft_consensus.cc:482] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:15.174489 10852 raft_consensus.cc:3037] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:15.177565 10852 raft_consensus.cc:504] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.178364 10852 leader_election.cc:288] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 election: Requested vote from peers 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933), b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053)
I20260924 01:02:15.178930 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fad49a993547a5b59974d85455f0d7" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2"
I20260924 01:02:15.178942 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fad49a993547a5b59974d85455f0d7" candidate_uuid: "4b249787f0b8411bb1b8aaef68e51c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452ebaca064d2ab2e184f968ca6d9b"
I20260924 01:02:15.179162 10774 raft_consensus.cc:3037] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:15.179190 10499 raft_consensus.cc:3037] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:15.181492 10499 raft_consensus.cc:2454] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b249787f0b8411bb1b8aaef68e51c80 in term 1.
I20260924 01:02:15.181953 10567 leader_election.cc:302] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392d78cfc2a8429c8f07cd539e1bd4b2, 4b249787f0b8411bb1b8aaef68e51c80; no voters:
I20260924 01:02:15.182269 10852 raft_consensus.cc:2781] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:15.182274 10774 raft_consensus.cc:2454] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b249787f0b8411bb1b8aaef68e51c80 in term 1.
I20260924 01:02:15.186062 10852 raft_consensus.cc:686] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 LEADER]: Becoming Leader. State: Replica: 4b249787f0b8411bb1b8aaef68e51c80, State: Running, Role: LEADER
I20260924 01:02:15.186604 10852 consensus_queue.cc:237] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.190373 10337 catalog_manager.cc:5495] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "4b249787f0b8411bb1b8aaef68e51c80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } health_report { overall_health: UNKNOWN } } }
Sep 24 01:02:15 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211732, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:15.267107 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.252752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.8.254.126:45723)
0924 01:02:15.253462 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.253473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.253697 (+ 224us) server_negotiation.cc:407] Connection header received
0924 01:02:15.253909 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.253918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.253971 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.254082 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:15.255240 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.255986 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.257418 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.257920 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.258055 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.261481 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:15.261486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:15.261489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:15.261517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:15.264254 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.264977 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.264990 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.264993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.265105 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.265475 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.265482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.265485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.265757 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:15.265918 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.266404 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.266745 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":538,"thread_start_us":368,"threads_started":1}
I20260924 01:02:15.267160 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.252752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45723 (local address 127.0.0.1:40302)
0924 01:02:15.253593 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:15.253608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:15.253784 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:15.254108 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:15.254112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:15.254866 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:15.255072 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.255082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.256133 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.256138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.257261 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.257274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.258103 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.258109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.258316 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.259017 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:15.259035 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:15.261273 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:15.264385 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:15.264401 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:15.264410 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:15.264849 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:15.265215 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:15.265221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:15.265224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:15.265369 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:15.265911 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:15.265919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:15.266074 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 01:02:15.266320 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 01:02:15.266636 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":4,"thread_start_us":583,"threads_started":1}
I20260924 01:02:15.273357 10337 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:40302:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 01:02:15.274497 10337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:15.281939 10613 tablet_service.cc:1467] Processing CreateTablet for tablet 386bc785b86f40e3a946cf9179fb2a27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32064cc381094b0d928e7c5fbbede876]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:02:15.282122 10473 tablet_service.cc:1467] Processing CreateTablet for tablet 386bc785b86f40e3a946cf9179fb2a27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32064cc381094b0d928e7c5fbbede876]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:02:15.282518 10613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386bc785b86f40e3a946cf9179fb2a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:15.282467 10753 tablet_service.cc:1467] Processing CreateTablet for tablet 386bc785b86f40e3a946cf9179fb2a27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32064cc381094b0d928e7c5fbbede876]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:02:15.282725 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386bc785b86f40e3a946cf9179fb2a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:15.283004 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386bc785b86f40e3a946cf9179fb2a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:15.289763 10877 tablet_bootstrap.cc:492] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: Bootstrap starting.
I20260924 01:02:15.290833 10876 tablet_bootstrap.cc:492] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: Bootstrap starting.
I20260924 01:02:15.292079 10877 tablet_bootstrap.cc:654] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:15.292760 10876 tablet_bootstrap.cc:654] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:15.295771 10877 tablet_bootstrap.cc:492] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: No bootstrap required, opened a new log
I20260924 01:02:15.295987 10877 ts_tablet_manager.cc:1397] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260924 01:02:15.296718 10877 raft_consensus.cc:348] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.297026 10877 raft_consensus.cc:374] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:15.297114 10877 raft_consensus.cc:729] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b249787f0b8411bb1b8aaef68e51c80, State: Initialized, Role: FOLLOWER
I20260924 01:02:15.297415 10877 consensus_queue.cc:260] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.297891 10852 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.298802 10852 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.299288 10877 ts_tablet_manager.cc:1428] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 01:02:15.299659 10878 tablet_bootstrap.cc:492] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: Bootstrap starting.
I20260924 01:02:15.301556 10878 tablet_bootstrap.cc:654] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:15.308621 10876 tablet_bootstrap.cc:492] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: No bootstrap required, opened a new log
I20260924 01:02:15.308818 10876 ts_tablet_manager.cc:1397] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20260924 01:02:15.309666 10876 raft_consensus.cc:348] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.309999 10876 raft_consensus.cc:374] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:15.310093 10876 raft_consensus.cc:729] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392d78cfc2a8429c8f07cd539e1bd4b2, State: Initialized, Role: FOLLOWER
I20260924 01:02:15.310389 10876 consensus_queue.cc:260] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.310870 10868 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.311283 10868 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.311755 10876 ts_tablet_manager.cc:1428] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:15.315269 10878 tablet_bootstrap.cc:492] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: No bootstrap required, opened a new log
I20260924 01:02:15.315470 10878 ts_tablet_manager.cc:1397] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.001s
I20260924 01:02:15.316198 10878 raft_consensus.cc:348] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.316460 10878 raft_consensus.cc:374] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:15.316584 10878 raft_consensus.cc:729] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0452ebaca064d2ab2e184f968ca6d9b, State: Initialized, Role: FOLLOWER
I20260924 01:02:15.317008 10878 consensus_queue.cc:260] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.317651 10878 ts_tablet_manager.cc:1428] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:02:15.317528 10863 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.317984 10863 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.474840 10863 raft_consensus.cc:482] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:15.475101 10863 raft_consensus.cc:504] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.476642 10863 leader_election.cc:288] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933), 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677)
I20260924 01:02:15.490237 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.477208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:46335 (local address 127.8.254.65:35933)
0924 01:02:15.477443 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.477450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.477474 (+ 24us) server_negotiation.cc:407] Connection header received
0924 01:02:15.477764 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.477774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.477887 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.477976 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:15.478882 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.480429 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.483833 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.485686 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.485925 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.486046 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.489721 (+ 3675us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.489901 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:02:15.491039 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.476868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35933 (local address 127.8.254.67:46335)
0924 01:02:15.477377 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:15.477402 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:15.477530 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:15.478281 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:15.478289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:15.478329 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:15.478655 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.478671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.480691 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.480699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.483562 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.483585 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.486704 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.486714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.489347 (+ 2633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.489545 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 01:02:15.490441 (+ 896us) client_negotiation.cc:240] Negotiation successful
0924 01:02:15.490636 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":142,"thread_start_us":280,"threads_started":1}
I20260924 01:02:15.491914 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386bc785b86f40e3a946cf9179fb2a27" candidate_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" is_pre_election: true
I20260924 01:02:15.492208 10499 raft_consensus.cc:2454] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0452ebaca064d2ab2e184f968ca6d9b in term 0.
I20260924 01:02:15.492883 10707 leader_election.cc:302] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392d78cfc2a8429c8f07cd539e1bd4b2, b0452ebaca064d2ab2e184f968ca6d9b; no voters:
I20260924 01:02:15.493407 10863 raft_consensus.cc:2781] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:15.493544 10863 raft_consensus.cc:482] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:15.493654 10863 raft_consensus.cc:3037] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:15.495544 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.477536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:34677 (local address 127.8.254.67:39133)
0924 01:02:15.486592 (+ 9056us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:15.486616 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:15.486752 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:15.487306 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:15.487310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:15.487339 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:15.487656 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.487670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.489312 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.489347 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.492246 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.492263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.494572 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.494578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.494841 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.494876 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:02:15.494976 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 01:02:15.495131 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8915,"mutex_wait_us":24,"thread_start_us":8791,"threads_started":1}
I20260924 01:02:15.496191 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.477746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:39133 (local address 127.8.254.66:34677)
0924 01:02:15.486125 (+ 8379us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.486131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.486865 (+ 734us) server_negotiation.cc:407] Connection header received
0924 01:02:15.486944 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.486948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.487026 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.487121 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:15.487892 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.489152 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.492460 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.493828 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.493994 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.494068 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.495764 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.495913 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8232,"thread_start_us":8762,"threads_started":1}
I20260924 01:02:15.497049 10863 raft_consensus.cc:504] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.497687 10863 leader_election.cc:288] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [CANDIDATE]: Term 1 election: Requested vote from peers 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933), 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677)
I20260924 01:02:15.498521 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386bc785b86f40e3a946cf9179fb2a27" candidate_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2"
I20260924 01:02:15.498762 10499 raft_consensus.cc:3037] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:15.501725 10499 raft_consensus.cc:2454] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0452ebaca064d2ab2e184f968ca6d9b in term 1.
I20260924 01:02:15.502249 10707 leader_election.cc:302] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392d78cfc2a8429c8f07cd539e1bd4b2, b0452ebaca064d2ab2e184f968ca6d9b; no voters:
I20260924 01:02:15.502302 10638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386bc785b86f40e3a946cf9179fb2a27" candidate_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b249787f0b8411bb1b8aaef68e51c80" is_pre_election: true
I20260924 01:02:15.502463 10633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386bc785b86f40e3a946cf9179fb2a27" candidate_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b249787f0b8411bb1b8aaef68e51c80"
I20260924 01:02:15.502574 10638 raft_consensus.cc:2454] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0452ebaca064d2ab2e184f968ca6d9b in term 0.
I20260924 01:02:15.502882 10863 raft_consensus.cc:2781] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:15.506464 10863 raft_consensus.cc:686] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 LEADER]: Becoming Leader. State: Replica: b0452ebaca064d2ab2e184f968ca6d9b, State: Running, Role: LEADER
I20260924 01:02:15.506873 10863 consensus_queue.cc:237] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } }
I20260924 01:02:15.508986 10897 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: TxnStatusTablet state changed. Reason: New leader b0452ebaca064d2ab2e184f968ca6d9b. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.509389 10897 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:02:15.510375 10899 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 01:02:15.510594 10899 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 01:02:15.510807 10336 catalog_manager.cc:5495] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b reported cstate change: term changed from 0 to 1, leader changed from <none> to b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67). New cstate: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } health_report { overall_health: HEALTHY } } }
I20260924 01:02:15.546681 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.540659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39053 (local address 127.0.0.1:49104)
0924 01:02:15.540922 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:15.540946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:15.541104 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:15.541739 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:15.541743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:15.541757 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:15.541960 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.541979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.543158 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.543164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.544228 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.544240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.544985 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.544992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.545168 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.545224 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.546092 (+ 868us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.546113 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 01:02:15.546217 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 01:02:15.546365 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 01:02:15.546715 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.540691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.8.254.67:39053)
0924 01:02:15.541363 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.541367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.541381 (+ 14us) server_negotiation.cc:407] Connection header received
0924 01:02:15.541458 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.541461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.541542 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.541712 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:15.542103 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.543030 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.544367 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.544822 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.544945 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.545419 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.545975 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.546143 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.546263 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.546340 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":307,"threads_started":1}
I20260924 01:02:15.551486 10633 raft_consensus.cc:3037] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:15.551692 10499 raft_consensus.cc:1261] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Refusing update from remote peer b0452ebaca064d2ab2e184f968ca6d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:15.553943 10633 raft_consensus.cc:1261] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Refusing update from remote peer b0452ebaca064d2ab2e184f968ca6d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:15.554658 10897 consensus_queue.cc:1035] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:15.555039 10863 consensus_queue.cc:1035] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:15.558929 10852 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: TxnStatusTablet state changed. Reason: New leader b0452ebaca064d2ab2e184f968ca6d9b. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.559224 10852 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.567369 10863 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.567674 10863 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:02:15.569299 10852 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.569563 10852 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P 4b249787f0b8411bb1b8aaef68e51c80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.582610 10868 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: TxnStatusTablet state changed. Reason: New leader b0452ebaca064d2ab2e184f968ca6d9b. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.582947 10868 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.584800 10852 consensus_queue.cc:1035] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:15.591957 10897 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.592242 10897 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P b0452ebaca064d2ab2e184f968ca6d9b: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:02:15.594960 10882 consensus_queue.cc:1035] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:15.595664 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.578178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.8.254.66:34677)
0924 01:02:15.578455 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.578465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.586680 (+ 8215us) server_negotiation.cc:407] Connection header received
0924 01:02:15.587046 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.587055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.587176 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.587312 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:15.588088 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.588924 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.590267 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.590877 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.591002 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.593964 (+ 2962us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.594720 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.594853 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.595340 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.595421 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 01:02:15.597361 10868 tablet_replica.cc:439] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392d78cfc2a8429c8f07cd539e1bd4b2" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35933 } } peers { permanent_uuid: "4b249787f0b8411bb1b8aaef68e51c80" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34677 } } peers { permanent_uuid: "b0452ebaca064d2ab2e184f968ca6d9b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39053 } } }
I20260924 01:02:15.597604 10868 tablet_replica.cc:442] T 386bc785b86f40e3a946cf9179fb2a27 P 392d78cfc2a8429c8f07cd539e1bd4b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:15.623769 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.614184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:33133 (local address 127.8.254.67:39053)
0924 01:02:15.617460 (+ 3276us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.617466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.617485 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:02:15.617555 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.617563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.617638 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.617743 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:15.618468 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.619630 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.621645 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.622371 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.622510 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.623017 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.623145 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.623229 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.623379 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.623463 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3131}
I20260924 01:02:15.623888 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.614117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39053 (local address 127.8.254.66:33133)
0924 01:02:15.614686 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:15.614707 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:15.614845 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:15.617927 (+ 3082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:15.617933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:15.617955 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:15.618305 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.618319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.619812 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.619817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.621480 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.621496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.622553 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.622559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.622740 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.622813 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.623252 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.623264 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 01:02:15.623337 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 01:02:15.623483 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":268,"threads_started":1}
I20260924 01:02:15.670665 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.663042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:34059 (local address 127.8.254.66:34677)
0924 01:02:15.663248 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.663256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.663904 (+ 648us) server_negotiation.cc:407] Connection header received
0924 01:02:15.664088 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.664092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.664159 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.664240 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:15.664922 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.665863 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.668373 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.668809 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.668947 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.669870 (+ 923us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.670001 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.670104 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.670251 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.670334 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:02:15.670898 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.663032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:34677 (local address 127.8.254.67:34059)
0924 01:02:15.663796 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:15.663816 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:15.663963 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:15.664277 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:15.664312 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:15.664327 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:15.664720 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.664732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.666243 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.666249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.668224 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:15.668243 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.669026 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.669032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:15.669212 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.669297 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:15.670107 (+ 810us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:15.670121 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 01:02:15.670240 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 01:02:15.670386 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":391,"threads_started":1}
I20260924 01:02:15.713652 10336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:15.713944 10336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:15.717692 10336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6068efbbc6c468d91d8a33c5b48161a: Sending DeleteTablet for 3 replicas of tablet 51fad49a993547a5b59974d85455f0d7
I20260924 01:02:15.718724 10473 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fad49a993547a5b59974d85455f0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:15 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:47220
I20260924 01:02:15.718729 10613 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fad49a993547a5b59974d85455f0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:15 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:41892
I20260924 01:02:15.719827 10753 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fad49a993547a5b59974d85455f0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:15 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49090
I20260924 01:02:15.728219 10930 tablet_replica.cc:330] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: stopping tablet replica
I20260924 01:02:15.728480 10930 raft_consensus.cc:2229] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:15.728790 10930 raft_consensus.cc:2258] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:15.731364 10930 ts_tablet_manager.cc:1905] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:15.738698 10929 tablet_replica.cc:330] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: stopping tablet replica
I20260924 01:02:15.738941 10929 raft_consensus.cc:2229] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:15.739593 10929 raft_consensus.cc:2258] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:15.742769 10930 ts_tablet_manager.cc:1918] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:15.742931 10930 log.cc:1196] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/51fad49a993547a5b59974d85455f0d7
I20260924 01:02:15.743386 10930 ts_tablet_manager.cc:1939] T 51fad49a993547a5b59974d85455f0d7 P b0452ebaca064d2ab2e184f968ca6d9b: Deleting consensus metadata
I20260924 01:02:15.743391 10929 ts_tablet_manager.cc:1905] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:15.744603 10321 catalog_manager.cc:4841] TS b0452ebaca064d2ab2e184f968ca6d9b (127.8.254.67:39053): tablet 51fad49a993547a5b59974d85455f0d7 (table test-table [id=2fc8cd3a1b9d4f7aab51d56f26cfa4f0]) successfully deleted
I20260924 01:02:15.750408 10929 ts_tablet_manager.cc:1918] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:15.750564 10929 log.cc:1196] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/51fad49a993547a5b59974d85455f0d7
I20260924 01:02:15.750844 10928 tablet_replica.cc:330] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: stopping tablet replica
I20260924 01:02:15.751049 10929 ts_tablet_manager.cc:1939] T 51fad49a993547a5b59974d85455f0d7 P 4b249787f0b8411bb1b8aaef68e51c80: Deleting consensus metadata
I20260924 01:02:15.751147 10928 raft_consensus.cc:2229] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:15.751439 10928 raft_consensus.cc:2258] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:15.752319 10323 catalog_manager.cc:4841] TS 4b249787f0b8411bb1b8aaef68e51c80 (127.8.254.66:34677): tablet 51fad49a993547a5b59974d85455f0d7 (table test-table [id=2fc8cd3a1b9d4f7aab51d56f26cfa4f0]) successfully deleted
I20260924 01:02:15.752664 10928 ts_tablet_manager.cc:1905] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:15.760223 10928 ts_tablet_manager.cc:1918] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:15.760533 10928 log.cc:1196] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/51fad49a993547a5b59974d85455f0d7
I20260924 01:02:15.761104 10928 ts_tablet_manager.cc:1939] T 51fad49a993547a5b59974d85455f0d7 P 392d78cfc2a8429c8f07cd539e1bd4b2: Deleting consensus metadata
I20260924 01:02:15.762501 10323 catalog_manager.cc:4841] TS 392d78cfc2a8429c8f07cd539e1bd4b2 (127.8.254.65:35933): tablet 51fad49a993547a5b59974d85455f0d7 (table test-table [id=2fc8cd3a1b9d4f7aab51d56f26cfa4f0]) successfully deleted
Sep 24 01:02:15 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:15.774016 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.748464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.8.254.65:35933)
0924 01:02:15.748662 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.748670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.753280 (+ 4610us) server_negotiation.cc:407] Connection header received
0924 01:02:15.753812 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.753835 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.753925 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.754003 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:15.756206 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.757341 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.759333 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.760066 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.760167 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.766001 (+ 5834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:15.766018 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:15.766039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:15.766087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:15.770224 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.771340 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.771351 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.771355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.771548 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.772003 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.772009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.772013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.772504 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:15.772842 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.773193 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.773458 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 01:02:15.774804 10539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47244
I20260924 01:02:15.804402 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.778777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.8.254.126:45723)
0924 01:02:15.786505 (+ 7728us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.786511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.786538 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:02:15.786614 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.786620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.786709 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.786838 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:15.788820 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.790091 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.792107 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.792812 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.792961 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.796159 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:15.796167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:15.796172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:15.796219 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:15.800272 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.801230 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.801249 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.801253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.801440 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.801908 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.801913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.801917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.802331 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:15.803624 (+ 1293us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.803757 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.804003 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7504,"thread_start_us":7338,"threads_started":1}
W20260924 01:02:15.805027 10336 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304
I20260924 01:02:15.807679 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10408
I20260924 01:02:15.834154 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.810573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.8.254.65:35933)
0924 01:02:15.810754 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.810760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.818355 (+ 7595us) server_negotiation.cc:407] Connection header received
0924 01:02:15.818443 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.818448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.818533 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.818635 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:15.820345 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.821382 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.823168 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.823792 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.823885 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.827230 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:15.827249 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:15.827253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:15.827298 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:15.830791 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.831728 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.831734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.831737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.831890 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.832371 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.832378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.832380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.832756 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:15.833025 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.833584 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.833829 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 01:02:15.835546 10539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47260
W20260924 01:02:15.836155 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:15.836320 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 10408
I20260924 01:02:15.868736 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10548
Sep 24 01:02:15 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:15.889222 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.869573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.8.254.66:34677)
0924 01:02:15.869801 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.869809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.870007 (+ 198us) server_negotiation.cc:407] Connection header received
0924 01:02:15.870234 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.870248 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.870358 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.870492 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:15.872121 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.873287 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.875253 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.875982 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.876133 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.881501 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:15.881517 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:15.881537 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:15.881594 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:15.885805 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.886739 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.886749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.886753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.886937 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.887389 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.887394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.887398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.887937 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:15.888251 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.888592 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.888837 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 01:02:15.889986 10680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41898
W20260924 01:02:15.890779 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:15.890991 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 10548
I20260924 01:02:15.910640 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10689
Sep 24 01:02:15 dist-test-slave-5c7l krb5kdc[10292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:15.929272 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:15.911439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.8.254.67:39053)
0924 01:02:15.911623 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:15.911628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:15.911700 (+ 72us) server_negotiation.cc:407] Connection header received
0924 01:02:15.911911 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:15.911914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:15.912000 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:15.912089 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:15.913873 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.914760 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.916630 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:15.917112 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:15.917211 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:15.922848 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:15.922861 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:15.922876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:15.922925 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:15.926281 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.927045 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.927064 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.927067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.927233 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:15.927630 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:15.927637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:15.927642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:15.928013 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:15.928331 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:15.928665 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 01:02:15.928895 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 01:02:15.929950 10820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49110
W20260924 01:02:15.930727 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10689: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:15.930878 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 10689
I20260924 01:02:15.951509 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10305
I20260924 01:02:15.952687 10377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:16.068835 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 10305
2026-09-24T01:02:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5175 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:16 dist-test-slave-5c7l krb5kdc[10940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:16 dist-test-slave-5c7l krb5kdc[10940](info): set up 2 sockets
Sep 24 01:02:16 dist-test-slave-5c7l krb5kdc[10940](info): commencing operation
krb5kdc: starting...
W20260924 01:02:17.171531 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:17 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:17.367125 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44441
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:44441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:17.692076 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:17.692391 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:17.692571 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:17.699877 10953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:17.699993 10953 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 01:02:17.700093 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:17.700202 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:17.700271 10953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:17.700372 10953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:17.710410 10953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:44441
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:17.711743 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:17.713152 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:17.725361 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:17.725332 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:17.725384 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:17.726325 10953 server_base.cc:1034] running on GCE node
I20260924 01:02:17.727106 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:17.728598 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:17.729852 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790211737729798 us; error 37 us; skew 500 ppm
I20260924 01:02:17.730252 10953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:17 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211737, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:17.733347 10953 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:17.734891 10953 webserver.cc:460] Webserver started at http://127.8.254.126:39509/ using document root <none> and password file <none>
I20260924 01:02:17.735611 10953 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:17.735749 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:17.736071 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:17.739068 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "f2a37d0d921e409f93630dab1cbe1619"
format_stamp: "Formatted at 2026-09-24 01:02:17 on dist-test-slave-5c7l"
server_key: "df8ede594060eca8ca990e75a13d110d"
server_key_iv: "a6f97fef58446ce292296ce52f9652fe"
server_key_version: "encryptionkey@0"
I20260924 01:02:17.739851 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "f2a37d0d921e409f93630dab1cbe1619"
format_stamp: "Formatted at 2026-09-24 01:02:17 on dist-test-slave-5c7l"
server_key: "df8ede594060eca8ca990e75a13d110d"
server_key_iv: "a6f97fef58446ce292296ce52f9652fe"
server_key_version: "encryptionkey@0"
I20260924 01:02:17.744972 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:17.748405 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:17.749648 10953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:17.749833 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "f2a37d0d921e409f93630dab1cbe1619"
format_stamp: "Formatted at 2026-09-24 01:02:17 on dist-test-slave-5c7l"
server_key: "df8ede594060eca8ca990e75a13d110d"
server_key_iv: "a6f97fef58446ce292296ce52f9652fe"
server_key_version: "encryptionkey@0"
I20260924 01:02:17.750061 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:17.770618 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:17.775427 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:17.775718 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:17.797940 10953 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:44441
I20260924 01:02:17.797963 11029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:44441 every 8 connection(s)
I20260924 01:02:17.799397 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:17.803157 11030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:17.805573 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 10953
I20260924 01:02:17.805789 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:17.806511 9209 external_mini_cluster.cc:1428] Setting key f5a4f4736a4ac682e0b3245f8b173b27
I20260924 01:02:17.815977 11030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Bootstrap starting.
I20260924 01:02:17.819754 11030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:17.821059 11030 log.cc:824] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Log is configured to *not* fsync() on all Append() calls
Sep 24 01:02:17 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:17.824254 11030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: No bootstrap required, opened a new log
I20260924 01:02:17.831168 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:17.808561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.8.254.126:44441)
0924 01:02:17.809337 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:17.809352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:17.809400 (+ 48us) server_negotiation.cc:407] Connection header received
0924 01:02:17.811786 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:17.811820 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:17.812343 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:17.812794 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:17.814044 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:17.815974 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:17.817777 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:17.818514 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:17.818642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:17.823666 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:17.823674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:17.823689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:17.823724 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:17.826969 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:17.827821 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:17.827828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:17.827837 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:17.828002 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:17.828395 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:17.828398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:17.828400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:17.828944 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:17.829165 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:17.829641 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 01:02:17.830033 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":321,"threads_started":1}
I20260924 01:02:17.833508 11030 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a37d0d921e409f93630dab1cbe1619" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44441 } }
I20260924 01:02:17.833917 11030 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:17.834044 11030 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a37d0d921e409f93630dab1cbe1619, State: Initialized, Role: FOLLOWER
I20260924 01:02:17.834653 11030 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a37d0d921e409f93630dab1cbe1619" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44441 } }
I20260924 01:02:17.834888 11030 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:17.835026 11030 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:17.835209 11030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:17.837339 11030 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a37d0d921e409f93630dab1cbe1619" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44441 } }
I20260924 01:02:17.837863 11030 leader_election.cc:302] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a37d0d921e409f93630dab1cbe1619; no voters:
I20260924 01:02:17.838625 11030 leader_election.cc:288] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:17.838773 11035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:17.839562 11035 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [term 1 LEADER]: Becoming Leader. State: Replica: f2a37d0d921e409f93630dab1cbe1619, State: Running, Role: LEADER
I20260924 01:02:17.840152 11035 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a37d0d921e409f93630dab1cbe1619" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44441 } }
I20260924 01:02:17.840956 11030 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:17.842887 11036 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2a37d0d921e409f93630dab1cbe1619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a37d0d921e409f93630dab1cbe1619" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44441 } } }
I20260924 01:02:17.843333 11036 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:17.843056 11037 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a37d0d921e409f93630dab1cbe1619. Latest consensus state: current_term: 1 leader_uuid: "f2a37d0d921e409f93630dab1cbe1619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a37d0d921e409f93630dab1cbe1619" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44441 } } }
I20260924 01:02:17.843586 11037 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:17.845536 11043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:17.851042 11043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:17.860770 11043 catalog_manager.cc:1347] Generated new cluster ID: 8345bc79466346c195a316e9cd7967c0
I20260924 01:02:17.860913 11043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:17.871433 11043 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:17.872725 11043 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:17.878716 11043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Generated new TSK 0
I20260924 01:02:17.879652 11043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:17.934371 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44441
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:18.264884 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:18.265169 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:18.265369 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:18.272670 11056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:18.272838 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:18.272980 11056 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 01:02:18.273072 11056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:02:18.273211 11056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:18.283180 11056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:44441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:18.284574 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:18.285959 11056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:18.299413 11061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:18.299369 11062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:18.299459 11064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:18.300247 11056 server_base.cc:1034] running on GCE node
I20260924 01:02:18.300884 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:18.301812 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:18.303087 11056 hybrid_clock.cc:648] HybridClock initialized: now 1790211738303032 us; error 41 us; skew 500 ppm
I20260924 01:02:18.303448 11056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:18 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211738, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:18.312716 11056 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:18.314386 11056 webserver.cc:460] Webserver started at http://127.8.254.65:45723/ using document root <none> and password file <none>
I20260924 01:02:18.315092 11056 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:18.315240 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:18.315563 11056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:18.318701 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "baa5e7f53efc428ba677843d65f31fa7"
format_stamp: "Formatted at 2026-09-24 01:02:18 on dist-test-slave-5c7l"
server_key: "ad3557733453cc5257b02f14c3570a64"
server_key_iv: "8c853506065fa1267ee22b1ddd9c98bd"
server_key_version: "encryptionkey@0"
I20260924 01:02:18.319459 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "baa5e7f53efc428ba677843d65f31fa7"
format_stamp: "Formatted at 2026-09-24 01:02:18 on dist-test-slave-5c7l"
server_key: "ad3557733453cc5257b02f14c3570a64"
server_key_iv: "8c853506065fa1267ee22b1ddd9c98bd"
server_key_version: "encryptionkey@0"
I20260924 01:02:18.325145 11056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:18.328671 11071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:18.330118 11056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:02:18.330339 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "baa5e7f53efc428ba677843d65f31fa7"
format_stamp: "Formatted at 2026-09-24 01:02:18 on dist-test-slave-5c7l"
server_key: "ad3557733453cc5257b02f14c3570a64"
server_key_iv: "8c853506065fa1267ee22b1ddd9c98bd"
server_key_version: "encryptionkey@0"
I20260924 01:02:18.330556 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:18.353243 11056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:18.357774 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:18.358083 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:18.360987 11056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:18.361126 11056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:18.361259 11056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:18.361395 11056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:02:18 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211738, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:18.385394 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.361405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:42821 (local address 127.8.254.126:44441)
0924 01:02:18.361991 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:18.362000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:18.363176 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 01:02:18.365504 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:18.365509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:18.365602 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:18.365732 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:18.368298 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.369385 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.372105 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.372721 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.372853 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.377527 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:18.377532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:18.377536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:18.377603 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:18.381308 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.382440 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.382449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.382456 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.382639 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.383102 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.383110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.383116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.383599 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:18.383788 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:18.384488 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 01:02:18.384811 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":311,"threads_started":1}
I20260924 01:02:18.386032 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.362170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.8.254.65:42821)
0924 01:02:18.363015 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:18.363056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:18.365148 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:18.366083 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:18.366100 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:18.367037 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:18.368005 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.368043 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.369597 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.369604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.371958 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.371970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.373033 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.373044 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.373259 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.374115 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:18.374143 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:18.377218 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:18.381529 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.381541 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.381567 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.382274 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.382773 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.382778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.382781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.382970 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.383733 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:18.383739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:18.384087 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 01:02:18.384326 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 01:02:18.384821 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":333,"threads_started":1}
I20260924 01:02:18.407397 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.393035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.8.254.65:44279)
0924 01:02:18.393636 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:18.393658 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:18.393830 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:18.394161 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:18.394165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:18.394827 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:18.395114 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.395123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.396117 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.396122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.397109 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.397121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.397916 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.397922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.398097 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.398827 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:18.398858 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:18.400226 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:18.403506 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.403512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.403516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.403947 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.404567 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.404575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.404582 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.404819 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.405781 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:18.405789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:18.406068 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 01:02:18.406295 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 01:02:18.406748 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":347,"threads_started":1}
I20260924 01:02:18.407552 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.392837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:44279 (local address 127.8.254.126:44441)
0924 01:02:18.393028 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:18.393032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:18.393749 (+ 717us) server_negotiation.cc:407] Connection header received
0924 01:02:18.393955 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:18.393960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:18.394033 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:18.394135 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:18.395265 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.395980 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.397327 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.397728 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.397873 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.400455 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:18.400461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:18.400467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:18.400506 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:18.403373 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.404075 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.404081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.404084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.404299 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.405036 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.405045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.405051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.405525 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:18.405703 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:18.406543 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 01:02:18.406928 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:02:18.425037 11056 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:34959
I20260924 01:02:18.425076 11193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:34959 every 8 connection(s)
I20260924 01:02:18.426422 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:18.428606 11194 heartbeater.cc:347] Connected to a master server at 127.8.254.126:44441
I20260924 01:02:18.428845 11194 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:18.429508 11194 heartbeater.cc:511] Master 127.8.254.126:44441 requested a full tablet report, sending...
I20260924 01:02:18.431489 10985 ts_manager.cc:194] Registered new tserver with Master: baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959)
I20260924 01:02:18.432950 10985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:42821
I20260924 01:02:18.435771 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 11056
I20260924 01:02:18.435972 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:18.436620 9209 external_mini_cluster.cc:1428] Setting key 871f7d591e79e6787d9a053ee97d204e
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:18.478080 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44441
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:18.807653 11198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:18.807986 11198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:18.808171 11198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:18.815822 11198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:18.815968 11198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:18.816123 11198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 01:02:18.816210 11198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:02:18.816358 11198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:18.827302 11198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:44441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:18.828846 11198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:18.830273 11198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:18.844400 11203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:18.844323 11206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:18.844357 11204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:18.845264 11198 server_base.cc:1034] running on GCE node
I20260924 01:02:18.845935 11198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:18.847024 11198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:18.848232 11198 hybrid_clock.cc:648] HybridClock initialized: now 1790211738848183 us; error 49 us; skew 500 ppm
I20260924 01:02:18.848532 11198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:18 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211738, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:18.851701 11198 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:18.853322 11198 webserver.cc:460] Webserver started at http://127.8.254.66:40637/ using document root <none> and password file <none>
I20260924 01:02:18.854035 11198 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:18.854172 11198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:18.854574 11198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:18.857807 11198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "7e34471a22eb46589c377044ebe0b485"
format_stamp: "Formatted at 2026-09-24 01:02:18 on dist-test-slave-5c7l"
server_key: "5cffb3270474f1fa2bb46434df082655"
server_key_iv: "0aab6de92ec63b50902bc9ee4fedd7ed"
server_key_version: "encryptionkey@0"
I20260924 01:02:18.858919 11198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "7e34471a22eb46589c377044ebe0b485"
format_stamp: "Formatted at 2026-09-24 01:02:18 on dist-test-slave-5c7l"
server_key: "5cffb3270474f1fa2bb46434df082655"
server_key_iv: "0aab6de92ec63b50902bc9ee4fedd7ed"
server_key_version: "encryptionkey@0"
I20260924 01:02:18.864355 11198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:18.868108 11213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:18.869570 11198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:02:18.869756 11198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "7e34471a22eb46589c377044ebe0b485"
format_stamp: "Formatted at 2026-09-24 01:02:18 on dist-test-slave-5c7l"
server_key: "5cffb3270474f1fa2bb46434df082655"
server_key_iv: "0aab6de92ec63b50902bc9ee4fedd7ed"
server_key_version: "encryptionkey@0"
I20260924 01:02:18.869971 11198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:18.895869 11198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:18.900377 11198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:18.900689 11198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:18.903683 11198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:18.903808 11198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:18.903931 11198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:18.904031 11198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:02:18 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211738, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:18.927528 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.904363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:53687 (local address 127.8.254.126:44441)
0924 01:02:18.904587 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:18.904594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:18.906275 (+ 1681us) server_negotiation.cc:407] Connection header received
0924 01:02:18.908587 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:18.908592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:18.908689 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:18.908800 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:18.911227 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.912313 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.915155 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.915794 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.915966 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.920551 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:18.920556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:18.920560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:18.920597 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:18.923889 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.924863 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.924869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.924872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.925004 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.925468 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.925478 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.925480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.925787 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:18.925962 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:18.926932 (+ 970us) server_negotiation.cc:299] Negotiation successful
0924 01:02:18.927176 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 01:02:18.928812 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.905103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.8.254.66:53687)
0924 01:02:18.906013 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:18.906060 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:18.908266 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:18.909115 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:18.909132 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:18.910020 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:18.910982 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.911016 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.912587 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.912595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.914997 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.915010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.916107 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.916119 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.916393 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.917267 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:18.917311 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:18.920277 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:18.924199 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.924211 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.924231 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.924715 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.925134 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.925139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.925143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.925340 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.925926 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:18.925934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:18.926424 (+ 490us) client_negotiation.cc:769] Sending connection context
0924 01:02:18.926859 (+ 435us) client_negotiation.cc:240] Negotiation successful
0924 01:02:18.927479 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":405,"threads_started":1}
I20260924 01:02:18.957345 11198 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:37387
I20260924 01:02:18.957403 11333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:37387 every 8 connection(s)
I20260924 01:02:18.959223 11198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:18.961136 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.939483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54277 (local address 127.8.254.126:44441)
0924 01:02:18.939715 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:18.939721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:18.940513 (+ 792us) server_negotiation.cc:407] Connection header received
0924 01:02:18.940924 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:18.940933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:18.941027 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:18.941121 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:18.943233 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.944515 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.946793 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.947417 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.947579 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.950782 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:18.950791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:18.950799 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:18.950854 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:18.954818 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.958277 (+ 3459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.958287 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.958293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.958480 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:18.958955 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:18.958960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:18.958963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:18.959456 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:18.959597 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:18.960354 (+ 757us) server_negotiation.cc:299] Negotiation successful
0924 01:02:18.960675 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 01:02:18.962002 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:18.939507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.8.254.66:54277)
0924 01:02:18.940375 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:18.940406 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:18.940717 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:18.941413 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:18.941421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:18.942342 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:18.942980 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.943002 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.944745 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.944756 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.946611 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:18.946625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:18.947685 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:18.947696 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:18.947948 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:18.948849 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:18.948883 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:18.950522 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:18.957484 (+ 6962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.957489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.957493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.958072 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.958645 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:18.958653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:18.958659 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:18.958822 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:18.959687 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:18.959693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:18.959972 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 01:02:18.961362 (+ 1390us) client_negotiation.cc:240] Negotiation successful
0924 01:02:18.961680 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":480,"threads_started":1}
I20260924 01:02:18.962296 11334 heartbeater.cc:347] Connected to a master server at 127.8.254.126:44441
I20260924 01:02:18.962858 11334 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:18.963622 11334 heartbeater.cc:511] Master 127.8.254.126:44441 requested a full tablet report, sending...
I20260924 01:02:18.965086 10985 ts_manager.cc:194] Registered new tserver with Master: 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387)
I20260924 01:02:18.966094 10985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:53687
I20260924 01:02:18.967653 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 11198
I20260924 01:02:18.967830 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:18.968380 9209 external_mini_cluster.cc:1428] Setting key 76d5990d2e5edbd0019e4e1ef5220c7f
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:19.009888 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44441
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:19.337869 11338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:19.338165 11338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:19.338335 11338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:19.345824 11338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:19.346029 11338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:19.346174 11338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 01:02:19.346313 11338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:02:19.346446 11338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:19.356873 11338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:44441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:19.358156 11338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:19.359591 11338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:19.373570 11346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:19.373736 11344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:19.374408 11338 server_base.cc:1034] running on GCE node
W20260924 01:02:19.374501 11343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:19.375207 11338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:19.376220 11338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:19.377473 11338 hybrid_clock.cc:648] HybridClock initialized: now 1790211739377436 us; error 61 us; skew 500 ppm
I20260924 01:02:19.377816 11338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:19 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211739, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:19.381152 11338 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:19.382825 11338 webserver.cc:460] Webserver started at http://127.8.254.67:33351/ using document root <none> and password file <none>
I20260924 01:02:19.383538 11338 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:19.383684 11338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:19.384008 11338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:19.387145 11338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "38d938f9bc8f4833b39c157e31199876"
format_stamp: "Formatted at 2026-09-24 01:02:19 on dist-test-slave-5c7l"
server_key: "0082f091ccc8c464e96882abe9a2f14a"
server_key_iv: "77a9b7b49967c90b3a422afb7c889b71"
server_key_version: "encryptionkey@0"
I20260924 01:02:19.387972 11338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "38d938f9bc8f4833b39c157e31199876"
format_stamp: "Formatted at 2026-09-24 01:02:19 on dist-test-slave-5c7l"
server_key: "0082f091ccc8c464e96882abe9a2f14a"
server_key_iv: "77a9b7b49967c90b3a422afb7c889b71"
server_key_version: "encryptionkey@0"
I20260924 01:02:19.393337 11338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:19.396977 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:19.398468 11338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:19.398645 11338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "38d938f9bc8f4833b39c157e31199876"
format_stamp: "Formatted at 2026-09-24 01:02:19 on dist-test-slave-5c7l"
server_key: "0082f091ccc8c464e96882abe9a2f14a"
server_key_iv: "77a9b7b49967c90b3a422afb7c889b71"
server_key_version: "encryptionkey@0"
I20260924 01:02:19.398871 11338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:19.416795 11338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:19.421422 11338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:19.421732 11338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:19.425733 11338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:19.425879 11338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:19.426002 11338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:19.426083 11338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:19.435892 11194 heartbeater.cc:503] Master 127.8.254.126:44441 was elected leader, sending a full tablet report...
Sep 24 01:02:19 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211739, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:19.454550 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.425341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:40561 (local address 127.8.254.126:44441)
0924 01:02:19.425590 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.425599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.427536 (+ 1937us) server_negotiation.cc:407] Connection header received
0924 01:02:19.430139 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.430146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.430298 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.430423 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:19.433336 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.434830 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.439122 (+ 4292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.439743 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.439924 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.445734 (+ 5810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:19.445743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:19.445751 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:19.445807 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:19.449731 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.451123 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.451132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.451138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.451341 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.451955 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.451960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.451964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.452257 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:19.452423 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.453895 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.454188 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 01:02:19.457190 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.426252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.8.254.67:40561)
0924 01:02:19.427352 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.427409 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.429814 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.430843 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.430852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.431794 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:19.433050 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.433087 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.435035 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.435048 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.438855 (+ 3807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.438880 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.440252 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.440258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.440574 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.441510 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:19.441563 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:19.445438 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:19.449982 (+ 4544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:19.449995 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:19.450026 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:19.450942 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:19.451509 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:19.451514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:19.451519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:19.451816 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:19.452405 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:19.452413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:19.453049 (+ 636us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.453455 (+ 406us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.455358 (+ 1903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":486,"threads_started":1}
I20260924 01:02:19.469787 11338 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:46157
I20260924 01:02:19.469830 11473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:46157 every 8 connection(s)
I20260924 01:02:19.471227 11338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:19.474135 11474 heartbeater.cc:347] Connected to a master server at 127.8.254.126:44441
I20260924 01:02:19.474431 11474 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:19.475282 11474 heartbeater.cc:511] Master 127.8.254.126:44441 requested a full tablet report, sending...
I20260924 01:02:19.476845 10985 ts_manager.cc:194] Registered new tserver with Master: 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157)
I20260924 01:02:19.477947 10985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:40561
I20260924 01:02:19.478849 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 11338
I20260924 01:02:19.479036 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:19.479461 9209 external_mini_cluster.cc:1428] Setting key 2aa8dabbe6e2ee4ec342a881c388db60
I20260924 01:02:19.479952 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.465235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:39181 (local address 127.8.254.126:44441)
0924 01:02:19.465423 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.465429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.465788 (+ 359us) server_negotiation.cc:407] Connection header received
0924 01:02:19.466130 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.466138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.466287 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.466412 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:19.467808 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.468514 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.470231 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.470660 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.470808 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.473544 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:19.473552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:19.473557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:19.473604 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:19.476836 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.477648 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.477653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.477655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.477796 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.478416 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.478423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.478429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.478736 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:19.478905 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.479377 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.479591 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:02:19.480042 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.464708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.8.254.67:39181)
0924 01:02:19.465652 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.465682 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.465933 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.466439 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.466443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.467186 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:19.467660 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.467672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.468649 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.468658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.470044 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.470060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.470850 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.470862 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.471048 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.471767 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:19.471788 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:19.473066 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:19.476966 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:19.476972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:19.476975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:19.477500 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:19.477944 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:19.477949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:19.477952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:19.478253 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:19.478877 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:19.478883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:19.479080 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.479271 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.479598 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":718,"threads_started":1}
I20260924 01:02:19.482765 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 01:02:19 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 01:02:19 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:19.533295 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.518489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.8.254.126:44441)
0924 01:02:19.518726 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.518733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.518939 (+ 206us) server_negotiation.cc:407] Connection header received
0924 01:02:19.519216 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.519220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.519287 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.519392 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:19.520762 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.521520 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.522855 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.523291 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.523436 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.527564 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:19.527570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:19.527573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:19.527615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:19.530043 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.530854 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.530860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.530864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.531277 (+ 413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.531758 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.531763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.531767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.532084 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:19.532346 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.532715 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.532900 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:02:19.537494 10985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:19.544576 10985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:19.569862 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.557459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.0.0.1:49760)
0924 01:02:19.559523 (+ 2064us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.559575 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.559863 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.561800 (+ 1937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.561807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.561840 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:19.562246 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.562263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.564182 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.564189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.566929 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.566944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.568808 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.568817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.569031 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.569101 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.569210 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.569431 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1688,"mutex_wait_us":59,"thread_start_us":1441,"threads_started":1}
I20260924 01:02:19.570959 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.557694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.8.254.66:37387)
0924 01:02:19.561116 (+ 3422us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.561124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.561163 (+ 39us) server_negotiation.cc:407] Connection header received
0924 01:02:19.561274 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.561284 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.561550 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.562317 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:19.562540 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.564027 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.567123 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.568576 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.570126 (+ 1550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.570416 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.570490 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.570660 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3257,"thread_start_us":4124,"threads_started":1}
I20260924 01:02:19.571274 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.558127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.8.254.65:34959)
0924 01:02:19.560711 (+ 2584us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.560722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.561106 (+ 384us) server_negotiation.cc:407] Connection header received
0924 01:02:19.561439 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.561446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.561689 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.561879 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:19.563066 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.564257 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.568288 (+ 4031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.569629 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.569764 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.569967 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.570822 (+ 855us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.570996 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2324,"thread_start_us":2121,"threads_started":1}
I20260924 01:02:19.571578 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.561763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:46157 (local address 127.0.0.1:49920)
0924 01:02:19.562898 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.562916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.563018 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.564134 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.564138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.564169 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:19.564706 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.564723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.566863 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.566868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.568922 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.568937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.570746 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.570753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.570935 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.570975 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.571092 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.571237 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":854,"threads_started":1}
I20260924 01:02:19.571980 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.557667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.0.0.1:46944)
0924 01:02:19.560999 (+ 3332us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.561018 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.561223 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.562213 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.562220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.562253 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:19.562831 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.562849 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.564453 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.564459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.568123 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.568135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.570412 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.570418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.570579 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.570613 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.571549 (+ 936us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.571699 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3180,"mutex_wait_us":36,"thread_start_us":3038,"threads_started":1}
I20260924 01:02:19.572507 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.561417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.8.254.67:46157)
0924 01:02:19.563580 (+ 2163us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.563588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.563623 (+ 35us) server_negotiation.cc:407] Connection header received
0924 01:02:19.563709 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.563717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.563926 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.564125 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:19.564961 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.566712 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.569099 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.570529 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.570686 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.570910 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.571850 (+ 940us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.572024 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1994,"thread_start_us":1830,"threads_started":1}
I20260924 01:02:19.575289 11264 tablet_service.cc:1467] Processing CreateTablet for tablet d1c1d4ad887c48169c64ff4b6d503b72 (DEFAULT_TABLE table=test-table [id=1b94d11f6616441998f21a033b83bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:19.575687 11124 tablet_service.cc:1467] Processing CreateTablet for tablet d1c1d4ad887c48169c64ff4b6d503b72 (DEFAULT_TABLE table=test-table [id=1b94d11f6616441998f21a033b83bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:19.576117 11403 tablet_service.cc:1467] Processing CreateTablet for tablet d1c1d4ad887c48169c64ff4b6d503b72 (DEFAULT_TABLE table=test-table [id=1b94d11f6616441998f21a033b83bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:19.576869 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c1d4ad887c48169c64ff4b6d503b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:19.576968 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c1d4ad887c48169c64ff4b6d503b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:19.577481 11403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c1d4ad887c48169c64ff4b6d503b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:19.589979 11493 tablet_bootstrap.cc:492] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Bootstrap starting.
I20260924 01:02:19.593263 11493 tablet_bootstrap.cc:654] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:19.594400 11493 log.cc:824] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:19.595399 11495 tablet_bootstrap.cc:492] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Bootstrap starting.
I20260924 01:02:19.596055 11494 tablet_bootstrap.cc:492] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Bootstrap starting.
I20260924 01:02:19.598616 11493 tablet_bootstrap.cc:492] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: No bootstrap required, opened a new log
I20260924 01:02:19.598906 11493 ts_tablet_manager.cc:1397] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 01:02:19.599730 11494 tablet_bootstrap.cc:654] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:19.599944 11495 tablet_bootstrap.cc:654] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:19.601080 11495 log.cc:824] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:19.601068 11494 log.cc:824] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:19.604084 11494 tablet_bootstrap.cc:492] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: No bootstrap required, opened a new log
I20260924 01:02:19.604308 11495 tablet_bootstrap.cc:492] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: No bootstrap required, opened a new log
I20260924 01:02:19.604398 11494 ts_tablet_manager.cc:1397] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 01:02:19.604575 11495 ts_tablet_manager.cc:1397] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 01:02:19.608814 11493 raft_consensus.cc:348] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.609300 11493 raft_consensus.cc:374] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:19.609404 11493 raft_consensus.cc:729] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7f53efc428ba677843d65f31fa7, State: Initialized, Role: FOLLOWER
I20260924 01:02:19.610054 11493 consensus_queue.cc:260] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.611876 11493 ts_tablet_manager.cc:1428] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 01:02:19.613125 11494 raft_consensus.cc:348] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.613481 11494 raft_consensus.cc:374] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:19.613606 11494 raft_consensus.cc:729] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34471a22eb46589c377044ebe0b485, State: Initialized, Role: FOLLOWER
I20260924 01:02:19.614305 11494 consensus_queue.cc:260] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.615331 11334 heartbeater.cc:503] Master 127.8.254.126:44441 was elected leader, sending a full tablet report...
I20260924 01:02:19.615116 11495 raft_consensus.cc:348] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.615586 11495 raft_consensus.cc:374] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:19.615731 11495 raft_consensus.cc:729] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Initialized, Role: FOLLOWER
I20260924 01:02:19.616360 11495 consensus_queue.cc:260] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.617360 11494 ts_tablet_manager.cc:1428] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 01:02:19.618829 11474 heartbeater.cc:503] Master 127.8.254.126:44441 was elected leader, sending a full tablet report...
I20260924 01:02:19.619436 11495 ts_tablet_manager.cc:1428] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
W20260924 01:02:19.678975 11195 tablet.cc:2369] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:19.713266 11335 tablet.cc:2369] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:19.723842 11475 tablet.cc:2369] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:02:19.759781 11501 raft_consensus.cc:482] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:19.760135 11501 raft_consensus.cc:504] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.762058 11501 leader_election.cc:288] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387), baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959)
I20260924 01:02:19.767901 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.762997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:39711 (local address 127.8.254.65:34959)
0924 01:02:19.763191 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.763195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.763211 (+ 16us) server_negotiation.cc:407] Connection header received
0924 01:02:19.763279 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.763283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.763345 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.763451 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:19.763875 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.764584 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.766182 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.766979 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.767149 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.767228 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.767530 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.767664 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:02:19.767936 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.762534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.8.254.67:39711)
0924 01:02:19.762795 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.762818 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.762978 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.763477 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.763482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.763513 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:19.763722 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.763730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.764732 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.764744 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.766010 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.766026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.767139 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.767145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.767363 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.767388 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.767500 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.767640 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 01:02:19.768946 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.762547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.8.254.67:34413)
0924 01:02:19.763499 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.763513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.763670 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.764008 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.764013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.764035 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:19.764356 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.764364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.765339 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.765345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.766781 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.766793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.768280 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.768288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.768468 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.768500 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.768596 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.768695 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":211,"thread_start_us":431,"threads_started":1}
I20260924 01:02:19.769073 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.762534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:34413 (local address 127.8.254.66:37387)
0924 01:02:19.762742 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.762747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.763607 (+ 860us) server_negotiation.cc:407] Connection header received
0924 01:02:19.763830 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.763834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.763892 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.763997 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:19.764504 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.765203 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.766941 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.767848 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.768028 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.768117 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.768655 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.768813 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 01:02:19.769109 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c1d4ad887c48169c64ff4b6d503b72" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7" is_pre_election: true
I20260924 01:02:19.769713 11149 raft_consensus.cc:2454] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 0.
I20260924 01:02:19.769927 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c1d4ad887c48169c64ff4b6d503b72" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485" is_pre_election: true
I20260924 01:02:19.770491 11288 raft_consensus.cc:2454] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 0.
I20260924 01:02:19.770738 11357 leader_election.cc:302] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, baa5e7f53efc428ba677843d65f31fa7; no voters:
I20260924 01:02:19.771493 11501 raft_consensus.cc:2781] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:19.771644 11501 raft_consensus.cc:482] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:19.771778 11501 raft_consensus.cc:3037] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:19.774041 11501 raft_consensus.cc:504] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.774667 11501 leader_election.cc:288] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 election: Requested vote from peers 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387), baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959)
I20260924 01:02:19.775218 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c1d4ad887c48169c64ff4b6d503b72" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7"
I20260924 01:02:19.775264 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c1d4ad887c48169c64ff4b6d503b72" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485"
I20260924 01:02:19.775449 11288 raft_consensus.cc:3037] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:19.775449 11149 raft_consensus.cc:3037] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:19.778719 11149 raft_consensus.cc:2454] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 1.
I20260924 01:02:19.778756 11288 raft_consensus.cc:2454] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 1.
I20260924 01:02:19.779253 11357 leader_election.cc:302] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, 7e34471a22eb46589c377044ebe0b485, baa5e7f53efc428ba677843d65f31fa7; no voters:
I20260924 01:02:19.779621 11501 raft_consensus.cc:2781] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:19.780081 11501 raft_consensus.cc:686] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 1 LEADER]: Becoming Leader. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Running, Role: LEADER
I20260924 01:02:19.780666 11501 consensus_queue.cc:237] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.785455 10985 catalog_manager.cc:5495] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d938f9bc8f4833b39c157e31199876 (127.8.254.67). New cstate: current_term: 1 leader_uuid: "38d938f9bc8f4833b39c157e31199876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } health_report { overall_health: HEALTHY } } }
Sep 24 01:02:19 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211737, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:19.829957 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.816820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44441 (local address 127.0.0.1:60632)
0924 01:02:19.817378 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:19.817393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:19.817560 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:19.817904 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:19.817908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:19.818691 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:19.818949 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.818958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.819965 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.819973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.820903 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:19.820915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.821655 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.821661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:19.821824 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.822519 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:19.822536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:19.824694 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:19.827549 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:19.827560 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:19.827566 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:19.828001 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:19.828405 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:19.828408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:19.828411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:19.828542 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:19.829079 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:19.829086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:19.829246 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 01:02:19.829389 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 01:02:19.829596 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260924 01:02:19.829974 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:19.816712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.8.254.126:44441)
0924 01:02:19.816898 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:19.816903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:19.817481 (+ 578us) server_negotiation.cc:407] Connection header received
0924 01:02:19.817694 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:19.817698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:19.817776 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:19.817880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:19.819092 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.819828 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.821053 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:19.821470 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:19.821612 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:19.824863 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:19.824868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:19.824871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:19.824901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:19.827411 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.828155 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.828159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.828161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.828310 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:19.828635 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:19.828638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:19.828640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:19.828918 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:19.829029 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:19.829481 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 01:02:19.829725 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":69}
I20260924 01:02:19.835407 10985 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 01:02:19.836510 10985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:19.843333 11124 tablet_service.cc:1467] Processing CreateTablet for tablet 019c13e8ce754f5e8a762ac2c419e8c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05cc39f0c6504533b38bba4609b9e00f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:02:19.843667 11403 tablet_service.cc:1467] Processing CreateTablet for tablet 019c13e8ce754f5e8a762ac2c419e8c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05cc39f0c6504533b38bba4609b9e00f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:02:19.844398 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019c13e8ce754f5e8a762ac2c419e8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:19.844427 11403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019c13e8ce754f5e8a762ac2c419e8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:19.851052 11264 tablet_service.cc:1467] Processing CreateTablet for tablet 019c13e8ce754f5e8a762ac2c419e8c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05cc39f0c6504533b38bba4609b9e00f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:02:19.851754 11493 tablet_bootstrap.cc:492] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: Bootstrap starting.
I20260924 01:02:19.851933 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019c13e8ce754f5e8a762ac2c419e8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:19.851998 11495 tablet_bootstrap.cc:492] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: Bootstrap starting.
I20260924 01:02:19.853446 11493 tablet_bootstrap.cc:654] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:19.853626 11495 tablet_bootstrap.cc:654] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:19.855912 11493 tablet_bootstrap.cc:492] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: No bootstrap required, opened a new log
I20260924 01:02:19.856070 11493 ts_tablet_manager.cc:1397] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 01:02:19.856101 11495 tablet_bootstrap.cc:492] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: No bootstrap required, opened a new log
I20260924 01:02:19.856545 11495 ts_tablet_manager.cc:1397] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 01:02:19.856904 11493 raft_consensus.cc:348] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.857167 11493 raft_consensus.cc:374] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:19.857280 11493 raft_consensus.cc:729] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7f53efc428ba677843d65f31fa7, State: Initialized, Role: FOLLOWER
I20260924 01:02:19.857323 11495 raft_consensus.cc:348] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.857546 11495 raft_consensus.cc:374] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:19.857636 11495 raft_consensus.cc:729] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Initialized, Role: FOLLOWER
I20260924 01:02:19.857529 11493 consensus_queue.cc:260] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.857887 11495 consensus_queue.cc:260] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.858517 11495 ts_tablet_manager.cc:1428] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:02:19.858479 11501 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:19.858785 11501 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:19.859108 11493 ts_tablet_manager.cc:1428] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 01:02:19.858036 11499 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:19.859491 11499 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:19.860970 11494 tablet_bootstrap.cc:492] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: Bootstrap starting.
I20260924 01:02:19.862604 11494 tablet_bootstrap.cc:654] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:19.864828 11494 tablet_bootstrap.cc:492] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: No bootstrap required, opened a new log
I20260924 01:02:19.864986 11494 ts_tablet_manager.cc:1397] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 01:02:19.865612 11494 raft_consensus.cc:348] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.865801 11494 raft_consensus.cc:374] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:19.865940 11494 raft_consensus.cc:729] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34471a22eb46589c377044ebe0b485, State: Initialized, Role: FOLLOWER
I20260924 01:02:19.866180 11494 consensus_queue.cc:260] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:19.866778 11494 ts_tablet_manager.cc:1428] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 01:02:19.866719 11500 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:19.867127 11500 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:20.032123 11499 raft_consensus.cc:482] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:20.032351 11499 raft_consensus.cc:504] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.034266 11499 leader_election.cc:288] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387), 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157)
I20260924 01:02:20.040532 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.034656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:48837 (local address 127.8.254.67:46157)
0924 01:02:20.034862 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:20.034870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:20.035288 (+ 418us) server_negotiation.cc:407] Connection header received
0924 01:02:20.035525 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:20.035531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:20.035599 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:20.035701 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:20.036251 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.036984 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.038785 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.039533 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.039724 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.039802 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:20.040138 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 01:02:20.040282 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 01:02:20.040962 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.034880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:53925 (local address 127.8.254.66:37387)
0924 01:02:20.035069 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:20.035076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:20.035823 (+ 747us) server_negotiation.cc:407] Connection header received
0924 01:02:20.036046 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:20.036052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:20.036148 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:20.036227 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:20.036726 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.037459 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.039177 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.039929 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.040139 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.040209 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:20.040553 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 01:02:20.040691 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 01:02:20.040999 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.034493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.8.254.65:53925)
0924 01:02:20.035708 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:20.035722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:20.035908 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:20.036254 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:20.036258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:20.036283 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:20.036565 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.036575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.037600 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.037607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.039024 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.039037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.040152 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.040158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.040343 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.040386 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 01:02:20.040484 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 01:02:20.040614 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":459,"thread_start_us":423,"threads_started":1}
I20260924 01:02:20.041285 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.034493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:46157 (local address 127.8.254.65:48837)
0924 01:02:20.035175 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:20.035193 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:20.035366 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:20.035725 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:20.035730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:20.035757 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:20.036086 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.036096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.037108 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.037115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.038626 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.038638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.039748 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.039753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.039913 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.039937 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 01:02:20.040794 (+ 857us) client_negotiation.cc:240] Negotiation successful
0924 01:02:20.040922 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":355,"threads_started":1}
I20260924 01:02:20.041638 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019c13e8ce754f5e8a762ac2c419e8c5" candidate_uuid: "baa5e7f53efc428ba677843d65f31fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485" is_pre_election: true
I20260924 01:02:20.041955 11288 raft_consensus.cc:2454] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5e7f53efc428ba677843d65f31fa7 in term 0.
I20260924 01:02:20.042253 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019c13e8ce754f5e8a762ac2c419e8c5" candidate_uuid: "baa5e7f53efc428ba677843d65f31fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d938f9bc8f4833b39c157e31199876" is_pre_election: true
I20260924 01:02:20.042536 11073 leader_election.cc:302] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e34471a22eb46589c377044ebe0b485, baa5e7f53efc428ba677843d65f31fa7; no voters:
I20260924 01:02:20.042642 11424 raft_consensus.cc:2454] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5e7f53efc428ba677843d65f31fa7 in term 0.
I20260924 01:02:20.042953 11499 raft_consensus.cc:2781] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:20.043120 11499 raft_consensus.cc:482] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:20.043234 11499 raft_consensus.cc:3037] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:20.045367 11499 raft_consensus.cc:504] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.045979 11499 leader_election.cc:288] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [CANDIDATE]: Term 1 election: Requested vote from peers 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387), 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157)
I20260924 01:02:20.046468 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019c13e8ce754f5e8a762ac2c419e8c5" candidate_uuid: "baa5e7f53efc428ba677843d65f31fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d938f9bc8f4833b39c157e31199876"
I20260924 01:02:20.046448 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019c13e8ce754f5e8a762ac2c419e8c5" candidate_uuid: "baa5e7f53efc428ba677843d65f31fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485"
I20260924 01:02:20.046663 11424 raft_consensus.cc:3037] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:20.046684 11288 raft_consensus.cc:3037] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:20.049712 11424 raft_consensus.cc:2454] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5e7f53efc428ba677843d65f31fa7 in term 1.
I20260924 01:02:20.049696 11288 raft_consensus.cc:2454] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5e7f53efc428ba677843d65f31fa7 in term 1.
I20260924 01:02:20.050186 11075 leader_election.cc:302] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, baa5e7f53efc428ba677843d65f31fa7; no voters:
I20260924 01:02:20.050558 11499 raft_consensus.cc:2781] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:20.050997 11499 raft_consensus.cc:686] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [term 1 LEADER]: Becoming Leader. State: Replica: baa5e7f53efc428ba677843d65f31fa7, State: Running, Role: LEADER
I20260924 01:02:20.051546 11499 consensus_queue.cc:237] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.055009 11516 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: TxnStatusTablet state changed. Reason: New leader baa5e7f53efc428ba677843d65f31fa7. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.055433 11516 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:02:20.056074 11518 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 01:02:20.056324 11518 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 01:02:20.056515 10984 catalog_manager.cc:5495] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65). New cstate: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20260924 01:02:20.103003 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.096496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.0.0.1:46946)
0924 01:02:20.096728 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:20.096751 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:20.096886 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:20.097480 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:20.097483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:20.097493 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:20.097648 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.097654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.099092 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.099096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.100171 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.100183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.101111 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.101115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.101281 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.101341 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.102415 (+ 1074us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.102444 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 01:02:20.102543 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 01:02:20.102690 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 01:02:20.103192 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.096637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.8.254.65:34959)
0924 01:02:20.096845 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:20.096850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:20.096870 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:02:20.097140 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:20.097146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:20.097284 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:20.097455 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:20.097776 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.098944 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.100313 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.100969 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.101097 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.101536 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.102295 (+ 759us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.102440 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:20.102574 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 01:02:20.102872 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 01:02:20.111508 11288 raft_consensus.cc:1261] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Refusing update from remote peer baa5e7f53efc428ba677843d65f31fa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:20.111550 11424 raft_consensus.cc:1261] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Refusing update from remote peer baa5e7f53efc428ba677843d65f31fa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:20.112469 11516 consensus_queue.cc:1035] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:20.112818 11499 consensus_queue.cc:1035] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:20.124509 11501 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: TxnStatusTablet state changed. Reason: New leader baa5e7f53efc428ba677843d65f31fa7. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.124845 11501 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:20.128611 11500 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: TxnStatusTablet state changed. Reason: New leader baa5e7f53efc428ba677843d65f31fa7. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.128921 11500 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:20.136992 11516 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.137442 11516 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:02:20.141309 11516 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.141665 11516 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P baa5e7f53efc428ba677843d65f31fa7: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:02:20.141716 11500 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.142002 11500 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P 7e34471a22eb46589c377044ebe0b485: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:20.142776 11501 tablet_replica.cc:439] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "baa5e7f53efc428ba677843d65f31fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } } }
I20260924 01:02:20.143070 11501 tablet_replica.cc:442] T 019c13e8ce754f5e8a762ac2c419e8c5 P 38d938f9bc8f4833b39c157e31199876: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:02:20.161872 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.155685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.8.254.67:46157)
0924 01:02:20.155917 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:20.155927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:20.156266 (+ 339us) server_negotiation.cc:407] Connection header received
0924 01:02:20.156531 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:20.156535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:20.156668 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:20.156845 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:20.157275 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.158114 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.159563 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.160021 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.160156 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.160610 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.161255 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.161421 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:20.161479 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 01:02:20.161564 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 01:02:20.178990 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.173318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:38419 (local address 127.8.254.65:34959)
0924 01:02:20.173553 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:20.173561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:20.173897 (+ 336us) server_negotiation.cc:407] Connection header received
0924 01:02:20.174178 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:20.174182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:20.174323 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:20.174423 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:20.174983 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.175750 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.177080 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.177529 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.177681 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.178153 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.178334 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.178435 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:20.178588 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 01:02:20.178669 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 01:02:20.179093 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.173191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.8.254.67:38419)
0924 01:02:20.173771 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:20.173787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:20.173998 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:20.174439 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:20.174443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:20.174454 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:20.174814 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.174823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.175878 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.175887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.176927 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.176938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.177711 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.177718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.177873 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.177927 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.178436 (+ 509us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.178456 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 01:02:20.178554 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 01:02:20.178690 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":261,"threads_started":1}
I20260924 01:02:20.188984 11288 raft_consensus.cc:1261] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Refusing update from remote peer 38d938f9bc8f4833b39c157e31199876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:20.189431 11149 raft_consensus.cc:1261] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Refusing update from remote peer 38d938f9bc8f4833b39c157e31199876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:20.189596 11503 consensus_queue.cc:1035] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:20.190176 11501 consensus_queue.cc:1035] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:20.198334 11522 mvcc.cc:205] Tried to move back new op lower bound from 7332707287807574016 to 7332707286146109440. Current Snapshot: MvccSnapshot[applied={T|T < 7332707287807574016}]
I20260924 01:02:20.229400 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.222355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:45969 (local address 127.8.254.67:46157)
0924 01:02:20.222555 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:20.222564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:20.223086 (+ 522us) server_negotiation.cc:407] Connection header received
0924 01:02:20.223403 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:20.223407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:20.223472 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:20.223552 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:20.224186 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.225182 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.227231 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.227995 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.228144 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.228559 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.228697 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.228802 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:20.228983 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 01:02:20.229081 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:02:20.229526 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:20.222288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:46157 (local address 127.8.254.65:45969)
0924 01:02:20.222974 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:20.222995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:20.223165 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:20.223598 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:20.223609 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:20.223623 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:20.224026 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.224039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.225371 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.225377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.227085 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:20.227098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:20.228179 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:20.228184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:20.228323 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:20.228381 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:20.228821 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:20.228843 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 01:02:20.228942 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 01:02:20.229087 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":446,"threads_started":1}
I20260924 01:02:20.273447 10984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:20.273752 10984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:20.277901 10984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Sending DeleteTablet for 3 replicas of tablet d1c1d4ad887c48169c64ff4b6d503b72
I20260924 01:02:20.279130 11264 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c1d4ad887c48169c64ff4b6d503b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:20 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49760
I20260924 01:02:20.279189 11124 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c1d4ad887c48169c64ff4b6d503b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:20 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:46944
I20260924 01:02:20.280445 11403 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c1d4ad887c48169c64ff4b6d503b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:20 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49920
I20260924 01:02:20.288834 10984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:20.290473 10984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:20.291930 11542 tablet_replica.cc:330] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: stopping tablet replica
I20260924 01:02:20.292534 11542 raft_consensus.cc:2229] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:20.292991 11542 raft_consensus.cc:2258] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:20.295101 11542 ts_tablet_manager.cc:1905] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:20.302784 11264 tablet_service.cc:1467] Processing CreateTablet for tablet 83062b7c96064c1fa6696382fe289e18 (DEFAULT_TABLE table=test-table [id=b550dcd4a06a496587e3cb7cbd32e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:20.303216 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83062b7c96064c1fa6696382fe289e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:20.306758 11402 tablet_service.cc:1467] Processing CreateTablet for tablet 83062b7c96064c1fa6696382fe289e18 (DEFAULT_TABLE table=test-table [id=b550dcd4a06a496587e3cb7cbd32e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:20.307269 11402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83062b7c96064c1fa6696382fe289e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:20.310855 11123 tablet_service.cc:1467] Processing CreateTablet for tablet 83062b7c96064c1fa6696382fe289e18 (DEFAULT_TABLE table=test-table [id=b550dcd4a06a496587e3cb7cbd32e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:20.311460 11123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83062b7c96064c1fa6696382fe289e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:20.325052 11544 tablet_replica.cc:330] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: stopping tablet replica
I20260924 01:02:20.325570 11544 raft_consensus.cc:2229] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:20.326294 11544 raft_consensus.cc:2258] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:20.326951 11494 tablet_bootstrap.cc:492] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Bootstrap starting.
I20260924 01:02:20.328760 11494 tablet_bootstrap.cc:654] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:20.329650 11544 ts_tablet_manager.cc:1905] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:20.330762 11543 tablet_replica.cc:330] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: stopping tablet replica
I20260924 01:02:20.331216 11543 raft_consensus.cc:2229] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:20.331707 11543 raft_consensus.cc:2258] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:20.332942 11495 tablet_bootstrap.cc:492] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Bootstrap starting.
I20260924 01:02:20.334868 11495 tablet_bootstrap.cc:654] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:20.336200 11543 ts_tablet_manager.cc:1905] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:20.343357 11493 tablet_bootstrap.cc:492] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Bootstrap starting.
I20260924 01:02:20.345268 11493 tablet_bootstrap.cc:654] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:20.353729 11543 ts_tablet_manager.cc:1918] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:20.354782 11542 ts_tablet_manager.cc:1918] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:20.354990 11542 log.cc:1196] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/d1c1d4ad887c48169c64ff4b6d503b72
I20260924 01:02:20.355482 11544 ts_tablet_manager.cc:1918] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:20.355753 11544 log.cc:1196] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/d1c1d4ad887c48169c64ff4b6d503b72
I20260924 01:02:20.356367 11544 ts_tablet_manager.cc:1939] T d1c1d4ad887c48169c64ff4b6d503b72 P 38d938f9bc8f4833b39c157e31199876: Deleting consensus metadata
I20260924 01:02:20.358320 11543 log.cc:1196] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/d1c1d4ad887c48169c64ff4b6d503b72
I20260924 01:02:20.358971 11543 ts_tablet_manager.cc:1939] T d1c1d4ad887c48169c64ff4b6d503b72 P baa5e7f53efc428ba677843d65f31fa7: Deleting consensus metadata
I20260924 01:02:20.358937 10972 catalog_manager.cc:4841] TS 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157): tablet d1c1d4ad887c48169c64ff4b6d503b72 (table test-table [id=1b94d11f6616441998f21a033b83bf74]) successfully deleted
I20260924 01:02:20.360881 11542 ts_tablet_manager.cc:1939] T d1c1d4ad887c48169c64ff4b6d503b72 P 7e34471a22eb46589c377044ebe0b485: Deleting consensus metadata
I20260924 01:02:20.362509 10972 catalog_manager.cc:4841] TS baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959): tablet d1c1d4ad887c48169c64ff4b6d503b72 (table test-table [id=1b94d11f6616441998f21a033b83bf74]) successfully deleted
I20260924 01:02:20.363900 10970 catalog_manager.cc:4841] TS 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387): tablet d1c1d4ad887c48169c64ff4b6d503b72 (table test-table [id=1b94d11f6616441998f21a033b83bf74]) successfully deleted
I20260924 01:02:20.367786 11493 tablet_bootstrap.cc:492] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: No bootstrap required, opened a new log
I20260924 01:02:20.367977 11493 ts_tablet_manager.cc:1397] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.008s
I20260924 01:02:20.368827 11493 raft_consensus.cc:348] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.369130 11493 raft_consensus.cc:374] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:20.369210 11493 raft_consensus.cc:729] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7f53efc428ba677843d65f31fa7, State: Initialized, Role: FOLLOWER
I20260924 01:02:20.369977 11493 consensus_queue.cc:260] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.370668 11494 tablet_bootstrap.cc:492] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: No bootstrap required, opened a new log
I20260924 01:02:20.370915 11494 ts_tablet_manager.cc:1397] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.012s
I20260924 01:02:20.371714 11495 tablet_bootstrap.cc:492] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: No bootstrap required, opened a new log
I20260924 01:02:20.371902 11495 ts_tablet_manager.cc:1397] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.004s
I20260924 01:02:20.371766 11494 raft_consensus.cc:348] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.372176 11494 raft_consensus.cc:374] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:20.372300 11494 raft_consensus.cc:729] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34471a22eb46589c377044ebe0b485, State: Initialized, Role: FOLLOWER
I20260924 01:02:20.372749 11495 raft_consensus.cc:348] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.373011 11495 raft_consensus.cc:374] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:20.373907 11493 ts_tablet_manager.cc:1428] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260924 01:02:20.373123 11495 raft_consensus.cc:729] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Initialized, Role: FOLLOWER
I20260924 01:02:20.374420 11495 consensus_queue.cc:260] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.374511 11494 consensus_queue.cc:260] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.376514 11494 ts_tablet_manager.cc:1428] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 01:02:20.376986 11495 ts_tablet_manager.cc:1428] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260924 01:02:20.500944 11539 raft_consensus.cc:482] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:20.501196 11539 raft_consensus.cc:504] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.502112 11539 leader_election.cc:288] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959), 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387)
I20260924 01:02:20.503088 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83062b7c96064c1fa6696382fe289e18" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7" is_pre_election: true
I20260924 01:02:20.503389 11149 raft_consensus.cc:2454] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 0.
I20260924 01:02:20.504256 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83062b7c96064c1fa6696382fe289e18" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485" is_pre_election: true
I20260924 01:02:20.504509 11288 raft_consensus.cc:2454] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 0.
I20260924 01:02:20.505096 11355 leader_election.cc:302] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, 7e34471a22eb46589c377044ebe0b485; no voters:
I20260924 01:02:20.505823 11539 raft_consensus.cc:2781] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:20.505956 11539 raft_consensus.cc:482] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:20.506063 11539 raft_consensus.cc:3037] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:20.510159 11539 raft_consensus.cc:504] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.512181 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83062b7c96064c1fa6696382fe289e18" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7"
I20260924 01:02:20.512424 11149 raft_consensus.cc:3037] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:20.515972 11149 raft_consensus.cc:2454] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 1.
I20260924 01:02:20.516465 11357 leader_election.cc:302] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, baa5e7f53efc428ba677843d65f31fa7; no voters:
I20260924 01:02:20.516776 11501 raft_consensus.cc:2781] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:20.516927 11501 raft_consensus.cc:686] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 1 LEADER]: Becoming Leader. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Running, Role: LEADER
I20260924 01:02:20.517204 11501 consensus_queue.cc:237] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:20.518862 11539 leader_election.cc:288] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 election: Requested vote from peers baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959), 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387)
I20260924 01:02:20.519461 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83062b7c96064c1fa6696382fe289e18" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485"
I20260924 01:02:20.519707 11288 raft_consensus.cc:3037] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:20.522174 10983 catalog_manager.cc:5495] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d938f9bc8f4833b39c157e31199876 (127.8.254.67). New cstate: current_term: 1 leader_uuid: "38d938f9bc8f4833b39c157e31199876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } health_report { overall_health: HEALTHY } } }
I20260924 01:02:20.522812 11288 raft_consensus.cc:2454] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 1.
I20260924 01:02:20.580123 11149 raft_consensus.cc:1261] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Refusing update from remote peer 38d938f9bc8f4833b39c157e31199876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:20.580678 11288 raft_consensus.cc:1261] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Refusing update from remote peer 38d938f9bc8f4833b39c157e31199876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:20.580718 11539 consensus_queue.cc:1035] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:20.581308 11539 consensus_queue.cc:1035] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 01:02:20.610025 11522 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 01:02:22.204995 10984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:22.205251 10984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:22.209250 10984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Sending DeleteTablet for 3 replicas of tablet 83062b7c96064c1fa6696382fe289e18
I20260924 01:02:22.210072 11123 tablet_service.cc:1514] Processing DeleteTablet for tablet 83062b7c96064c1fa6696382fe289e18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:22 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:46944
I20260924 01:02:22.210180 11264 tablet_service.cc:1514] Processing DeleteTablet for tablet 83062b7c96064c1fa6696382fe289e18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:22 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49760
I20260924 01:02:22.210266 11402 tablet_service.cc:1514] Processing DeleteTablet for tablet 83062b7c96064c1fa6696382fe289e18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:22 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49920
I20260924 01:02:22.221206 11577 tablet_replica.cc:330] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: stopping tablet replica
I20260924 01:02:22.221678 11577 raft_consensus.cc:2229] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:22.222070 11577 raft_consensus.cc:2258] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:22.223582 11577 ts_tablet_manager.cc:1905] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:22.229540 11576 tablet_replica.cc:330] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: stopping tablet replica
I20260924 01:02:22.229782 11576 raft_consensus.cc:2229] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:22.230019 11576 raft_consensus.cc:2258] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:22.231009 11576 ts_tablet_manager.cc:1905] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:22.233126 11577 ts_tablet_manager.cc:1918] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 01:02:22.233302 11577 log.cc:1196] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/83062b7c96064c1fa6696382fe289e18
I20260924 01:02:22.234336 11578 tablet_replica.cc:330] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: stopping tablet replica
I20260924 01:02:22.234566 11578 raft_consensus.cc:2229] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:22.235023 11578 raft_consensus.cc:2258] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:22.236028 11578 ts_tablet_manager.cc:1905] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:22.236850 11576 ts_tablet_manager.cc:1918] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 01:02:22.237674 11576 log.cc:1196] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/83062b7c96064c1fa6696382fe289e18
I20260924 01:02:22.238024 11577 ts_tablet_manager.cc:1939] T 83062b7c96064c1fa6696382fe289e18 P 7e34471a22eb46589c377044ebe0b485: Deleting consensus metadata
I20260924 01:02:22.238157 11576 ts_tablet_manager.cc:1939] T 83062b7c96064c1fa6696382fe289e18 P baa5e7f53efc428ba677843d65f31fa7: Deleting consensus metadata
I20260924 01:02:22.239472 10970 catalog_manager.cc:4841] TS 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387): tablet 83062b7c96064c1fa6696382fe289e18 (table test-table [id=b550dcd4a06a496587e3cb7cbd32e6e9]) successfully deleted
I20260924 01:02:22.239495 10972 catalog_manager.cc:4841] TS baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959): tablet 83062b7c96064c1fa6696382fe289e18 (table test-table [id=b550dcd4a06a496587e3cb7cbd32e6e9]) successfully deleted
I20260924 01:02:22.241829 11578 ts_tablet_manager.cc:1918] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 01:02:22.242012 11578 log.cc:1196] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/83062b7c96064c1fa6696382fe289e18
I20260924 01:02:22.242468 11578 ts_tablet_manager.cc:1939] T 83062b7c96064c1fa6696382fe289e18 P 38d938f9bc8f4833b39c157e31199876: Deleting consensus metadata
I20260924 01:02:22.243427 10972 catalog_manager.cc:4841] TS 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157): tablet 83062b7c96064c1fa6696382fe289e18 (table test-table [id=b550dcd4a06a496587e3cb7cbd32e6e9]) successfully deleted
Sep 24 01:02:22 dist-test-slave-5c7l krb5kdc[10940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 01:02:22 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:22.274447 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.259842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.8.254.126:44441)
0924 01:02:22.260491 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.260498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.260594 (+ 96us) server_negotiation.cc:407] Connection header received
0924 01:02:22.260836 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.260842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.260920 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.261047 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:22.262476 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.263269 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.264569 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.264997 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.265133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.269039 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:22.269044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:22.269047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:22.269125 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:22.271843 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:22.272469 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:22.272473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:22.272475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:22.272604 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:22.272977 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:22.272981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:22.272983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:22.273354 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:22.273608 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.273883 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.274052 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
I20260924 01:02:22.277849 10984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:22.278529 10984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:22.285036 11123 tablet_service.cc:1467] Processing CreateTablet for tablet faadd93d7849455393c53daca1ca4823 (DEFAULT_TABLE table=test-table [id=ae03ffcd871f484799be20d9fd07271b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:22.285254 11402 tablet_service.cc:1467] Processing CreateTablet for tablet faadd93d7849455393c53daca1ca4823 (DEFAULT_TABLE table=test-table [id=ae03ffcd871f484799be20d9fd07271b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:22.285600 11123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faadd93d7849455393c53daca1ca4823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:22.285771 11402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faadd93d7849455393c53daca1ca4823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:22.288424 11264 tablet_service.cc:1467] Processing CreateTablet for tablet faadd93d7849455393c53daca1ca4823 (DEFAULT_TABLE table=test-table [id=ae03ffcd871f484799be20d9fd07271b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:22.288887 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faadd93d7849455393c53daca1ca4823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:22.291040 11587 tablet_bootstrap.cc:492] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Bootstrap starting.
I20260924 01:02:22.292699 11588 tablet_bootstrap.cc:492] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Bootstrap starting.
I20260924 01:02:22.293085 11587 tablet_bootstrap.cc:654] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:22.294613 11588 tablet_bootstrap.cc:654] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:22.296001 11587 tablet_bootstrap.cc:492] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: No bootstrap required, opened a new log
I20260924 01:02:22.296305 11587 ts_tablet_manager.cc:1397] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 01:02:22.297015 11591 tablet_bootstrap.cc:492] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Bootstrap starting.
I20260924 01:02:22.297331 11588 tablet_bootstrap.cc:492] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: No bootstrap required, opened a new log
I20260924 01:02:22.297262 11587 raft_consensus.cc:348] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.297494 11588 ts_tablet_manager.cc:1397] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 01:02:22.297560 11587 raft_consensus.cc:374] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:22.297667 11587 raft_consensus.cc:729] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7f53efc428ba677843d65f31fa7, State: Initialized, Role: FOLLOWER
I20260924 01:02:22.297925 11587 consensus_queue.cc:260] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.298521 11588 raft_consensus.cc:348] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.298805 11587 ts_tablet_manager.cc:1428] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:02:22.298804 11588 raft_consensus.cc:374] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:22.298910 11588 raft_consensus.cc:729] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Initialized, Role: FOLLOWER
I20260924 01:02:22.299175 11588 consensus_queue.cc:260] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.299815 11588 ts_tablet_manager.cc:1428] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 01:02:22.300345 11591 tablet_bootstrap.cc:654] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:22.304052 11591 tablet_bootstrap.cc:492] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: No bootstrap required, opened a new log
I20260924 01:02:22.304224 11591 ts_tablet_manager.cc:1397] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 01:02:22.304877 11591 raft_consensus.cc:348] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.305117 11591 raft_consensus.cc:374] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:22.305212 11591 raft_consensus.cc:729] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34471a22eb46589c377044ebe0b485, State: Initialized, Role: FOLLOWER
I20260924 01:02:22.305467 11591 consensus_queue.cc:260] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.306331 11591 ts_tablet_manager.cc:1428] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:02:22.321792 11555 raft_consensus.cc:482] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:22.322010 11555 raft_consensus.cc:504] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.322700 11555 leader_election.cc:288] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959), 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387)
I20260924 01:02:22.323278 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faadd93d7849455393c53daca1ca4823" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485" is_pre_election: true
I20260924 01:02:22.323273 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faadd93d7849455393c53daca1ca4823" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7" is_pre_election: true
I20260924 01:02:22.323537 11288 raft_consensus.cc:2454] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 0.
I20260924 01:02:22.323565 11148 raft_consensus.cc:2454] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 0.
I20260924 01:02:22.324076 11355 leader_election.cc:302] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, 7e34471a22eb46589c377044ebe0b485; no voters:
I20260924 01:02:22.324429 11555 raft_consensus.cc:2781] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:22.324573 11555 raft_consensus.cc:482] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:22.324651 11555 raft_consensus.cc:3037] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:22.326922 11555 raft_consensus.cc:504] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.327456 11555 leader_election.cc:288] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 election: Requested vote from peers baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959), 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387)
I20260924 01:02:22.328009 11288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faadd93d7849455393c53daca1ca4823" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e34471a22eb46589c377044ebe0b485"
I20260924 01:02:22.328039 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faadd93d7849455393c53daca1ca4823" candidate_uuid: "38d938f9bc8f4833b39c157e31199876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7"
I20260924 01:02:22.328239 11288 raft_consensus.cc:3037] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:22.328254 11148 raft_consensus.cc:3037] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:22.331875 11288 raft_consensus.cc:2454] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 1.
I20260924 01:02:22.332302 11148 raft_consensus.cc:2454] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38d938f9bc8f4833b39c157e31199876 in term 1.
I20260924 01:02:22.332522 11355 leader_election.cc:302] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, 7e34471a22eb46589c377044ebe0b485; no voters:
I20260924 01:02:22.332938 11555 raft_consensus.cc:2781] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:22.333164 11555 raft_consensus.cc:686] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 1 LEADER]: Becoming Leader. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Running, Role: LEADER
I20260924 01:02:22.333513 11555 consensus_queue.cc:237] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.336784 10985 catalog_manager.cc:5495] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38d938f9bc8f4833b39c157e31199876 (127.8.254.67). New cstate: current_term: 1 leader_uuid: "38d938f9bc8f4833b39c157e31199876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } health_report { overall_health: HEALTHY } } }
I20260924 01:02:22.364711 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.359177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.8.254.67:46157)
0924 01:02:22.359786 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.359791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.359809 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:02:22.359860 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.359864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.359925 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.360035 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:22.360483 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.361251 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.362665 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.363271 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.363400 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.363871 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:22.363986 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:22.364062 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.364244 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.364349 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":300,"threads_started":1}
I20260924 01:02:22.374312 11288 raft_consensus.cc:1261] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Refusing update from remote peer 38d938f9bc8f4833b39c157e31199876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:22.374352 11148 raft_consensus.cc:1261] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Refusing update from remote peer 38d938f9bc8f4833b39c157e31199876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:22.374948 11556 consensus_queue.cc:1035] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:22.375260 11555 consensus_queue.cc:1035] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:22.419664 10983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:22.419955 10983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:22.423472 10983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Sending DeleteTablet for 3 replicas of tablet faadd93d7849455393c53daca1ca4823
I20260924 01:02:22.424211 11123 tablet_service.cc:1514] Processing DeleteTablet for tablet faadd93d7849455393c53daca1ca4823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:22 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:46944
I20260924 01:02:22.424264 11402 tablet_service.cc:1514] Processing DeleteTablet for tablet faadd93d7849455393c53daca1ca4823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:22 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49920
I20260924 01:02:22.424299 11264 tablet_service.cc:1514] Processing DeleteTablet for tablet faadd93d7849455393c53daca1ca4823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:22 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49760
I20260924 01:02:22.424501 11578 tablet_replica.cc:330] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: stopping tablet replica
I20260924 01:02:22.424500 11576 tablet_replica.cc:330] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: stopping tablet replica
I20260924 01:02:22.424700 11578 raft_consensus.cc:2229] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:22.424693 11576 raft_consensus.cc:2229] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:22.424894 11576 raft_consensus.cc:2258] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:22.425040 11578 raft_consensus.cc:2258] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:22.425197 11577 tablet_replica.cc:330] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: stopping tablet replica
I20260924 01:02:22.425391 11577 raft_consensus.cc:2229] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:22.425594 11577 raft_consensus.cc:2258] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:22.426162 11578 ts_tablet_manager.cc:1905] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:22.426667 11576 ts_tablet_manager.cc:1905] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:22.431357 11577 ts_tablet_manager.cc:1905] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:22.431185 10983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:22.431876 10983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:22.435071 11578 ts_tablet_manager.cc:1918] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:22.435236 11578 log.cc:1196] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/faadd93d7849455393c53daca1ca4823
I20260924 01:02:22.435627 11578 ts_tablet_manager.cc:1939] T faadd93d7849455393c53daca1ca4823 P 38d938f9bc8f4833b39c157e31199876: Deleting consensus metadata
I20260924 01:02:22.436798 10972 catalog_manager.cc:4841] TS 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157): tablet faadd93d7849455393c53daca1ca4823 (table test-table [id=ae03ffcd871f484799be20d9fd07271b]) successfully deleted
I20260924 01:02:22.439183 11576 ts_tablet_manager.cc:1918] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:22.439375 11576 log.cc:1196] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/faadd93d7849455393c53daca1ca4823
I20260924 01:02:22.440884 11264 tablet_service.cc:1467] Processing CreateTablet for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 (DEFAULT_TABLE table=test-table [id=8644edb4c93644cfbb22ef004be2cd5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:22.440898 11402 tablet_service.cc:1467] Processing CreateTablet for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 (DEFAULT_TABLE table=test-table [id=8644edb4c93644cfbb22ef004be2cd5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:22.441295 11402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:22.441319 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:22.443110 11577 ts_tablet_manager.cc:1918] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:02:22.443655 11576 ts_tablet_manager.cc:1939] T faadd93d7849455393c53daca1ca4823 P baa5e7f53efc428ba677843d65f31fa7: Deleting consensus metadata
I20260924 01:02:22.444355 11577 log.cc:1196] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/faadd93d7849455393c53daca1ca4823
I20260924 01:02:22.444900 11577 ts_tablet_manager.cc:1939] T faadd93d7849455393c53daca1ca4823 P 7e34471a22eb46589c377044ebe0b485: Deleting consensus metadata
I20260924 01:02:22.446038 10972 catalog_manager.cc:4841] TS baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959): tablet faadd93d7849455393c53daca1ca4823 (table test-table [id=ae03ffcd871f484799be20d9fd07271b]) successfully deleted
I20260924 01:02:22.446626 11123 tablet_service.cc:1467] Processing CreateTablet for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 (DEFAULT_TABLE table=test-table [id=8644edb4c93644cfbb22ef004be2cd5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:22.447074 11123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:22.447948 11591 tablet_bootstrap.cc:492] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Bootstrap starting.
I20260924 01:02:22.449635 11591 tablet_bootstrap.cc:654] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:22.450891 10970 catalog_manager.cc:4841] TS 7e34471a22eb46589c377044ebe0b485 (127.8.254.66:37387): tablet faadd93d7849455393c53daca1ca4823 (table test-table [id=ae03ffcd871f484799be20d9fd07271b]) successfully deleted
I20260924 01:02:22.454839 11587 tablet_bootstrap.cc:492] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: Bootstrap starting.
I20260924 01:02:22.454994 11591 tablet_bootstrap.cc:492] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: No bootstrap required, opened a new log
I20260924 01:02:22.455200 11591 ts_tablet_manager.cc:1397] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 01:02:22.455983 11591 raft_consensus.cc:348] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.456332 11591 raft_consensus.cc:374] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:22.456444 11591 raft_consensus.cc:729] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34471a22eb46589c377044ebe0b485, State: Initialized, Role: FOLLOWER
I20260924 01:02:22.456619 11587 tablet_bootstrap.cc:654] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:22.456790 11591 consensus_queue.cc:260] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.456979 11588 tablet_bootstrap.cc:492] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Bootstrap starting.
I20260924 01:02:22.457564 11591 ts_tablet_manager.cc:1428] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 01:02:22.458685 11588 tablet_bootstrap.cc:654] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:22.459756 11587 tablet_bootstrap.cc:492] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: No bootstrap required, opened a new log
I20260924 01:02:22.459934 11587 ts_tablet_manager.cc:1397] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 01:02:22.460736 11587 raft_consensus.cc:348] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.461009 11587 raft_consensus.cc:374] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:22.461117 11587 raft_consensus.cc:729] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7f53efc428ba677843d65f31fa7, State: Initialized, Role: FOLLOWER
I20260924 01:02:22.461439 11587 consensus_queue.cc:260] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.461993 11587 ts_tablet_manager.cc:1428] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 01:02:22.462733 11588 tablet_bootstrap.cc:492] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: No bootstrap required, opened a new log
I20260924 01:02:22.462914 11588 ts_tablet_manager.cc:1397] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 01:02:22.463549 11588 raft_consensus.cc:348] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.463799 11588 raft_consensus.cc:374] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:22.463891 11588 raft_consensus.cc:729] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d938f9bc8f4833b39c157e31199876, State: Initialized, Role: FOLLOWER
I20260924 01:02:22.464133 11588 consensus_queue.cc:260] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.464690 11588 ts_tablet_manager.cc:1428] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:02:22.536293 11593 raft_consensus.cc:482] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:22.536522 11593 raft_consensus.cc:504] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.538036 11593 leader_election.cc:288] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959), 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157)
I20260924 01:02:22.544730 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.538433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:39591 (local address 127.8.254.65:34959)
0924 01:02:22.539035 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.539041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.539071 (+ 30us) server_negotiation.cc:407] Connection header received
0924 01:02:22.539325 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.539330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.539405 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.539536 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:22.540115 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.540927 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.542947 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.543721 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.543917 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.543999 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.544263 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.544394 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":293,"threads_started":1}
I20260924 01:02:22.545135 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.539227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:38287 (local address 127.8.254.67:46157)
0924 01:02:22.539397 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.539404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.539676 (+ 272us) server_negotiation.cc:407] Connection header received
0924 01:02:22.539922 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.539927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.539994 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.540090 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:22.540590 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.541506 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.543265 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.544135 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.544342 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.544414 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.544746 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.544890 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:02:22.545181 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.538291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.8.254.66:39591)
0924 01:02:22.538937 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:22.538960 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:22.539116 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:22.539568 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:22.539573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:22.539602 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:22.539933 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.539942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.541076 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.541086 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.542790 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.542805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.543928 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.543934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.544108 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.544132 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 01:02:22.544238 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 01:02:22.544651 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":42,"spinlock_wait_cycles":11008,"thread_start_us":259,"threads_started":1}
I20260924 01:02:22.545863 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c88db6d8b4bd9ace2b3e6e35d49b1" candidate_uuid: "7e34471a22eb46589c377044ebe0b485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7" is_pre_election: true
I20260924 01:02:22.546020 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.538988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:46157 (local address 127.8.254.66:38287)
0924 01:02:22.539569 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:22.539583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:22.539766 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:22.540103 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:22.540107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:22.540129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:22.540447 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.540455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.541635 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.541641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.543118 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.543130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.544360 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.544366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.544553 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.544584 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 01:02:22.545409 (+ 825us) client_negotiation.cc:240] Negotiation successful
0924 01:02:22.545538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":304,"threads_started":1}
I20260924 01:02:22.546140 11148 raft_consensus.cc:2454] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e34471a22eb46589c377044ebe0b485 in term 0.
I20260924 01:02:22.546696 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c88db6d8b4bd9ace2b3e6e35d49b1" candidate_uuid: "7e34471a22eb46589c377044ebe0b485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d938f9bc8f4833b39c157e31199876" is_pre_election: true
I20260924 01:02:22.546801 11217 leader_election.cc:302] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e34471a22eb46589c377044ebe0b485, baa5e7f53efc428ba677843d65f31fa7; no voters:
I20260924 01:02:22.546965 11424 raft_consensus.cc:2454] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e34471a22eb46589c377044ebe0b485 in term 0.
I20260924 01:02:22.547272 11593 raft_consensus.cc:2781] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:22.547454 11593 raft_consensus.cc:482] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:22.547582 11593 raft_consensus.cc:3037] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:22.550122 11593 raft_consensus.cc:504] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.550815 11593 leader_election.cc:288] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [CANDIDATE]: Term 1 election: Requested vote from peers baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959), 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157)
I20260924 01:02:22.551270 11148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c88db6d8b4bd9ace2b3e6e35d49b1" candidate_uuid: "7e34471a22eb46589c377044ebe0b485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7f53efc428ba677843d65f31fa7"
I20260924 01:02:22.551370 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "769c88db6d8b4bd9ace2b3e6e35d49b1" candidate_uuid: "7e34471a22eb46589c377044ebe0b485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d938f9bc8f4833b39c157e31199876"
I20260924 01:02:22.551479 11148 raft_consensus.cc:3037] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:22.551638 11424 raft_consensus.cc:3037] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:22.553671 11424 raft_consensus.cc:2454] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e34471a22eb46589c377044ebe0b485 in term 1.
I20260924 01:02:22.554118 11217 leader_election.cc:302] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38d938f9bc8f4833b39c157e31199876, 7e34471a22eb46589c377044ebe0b485; no voters:
I20260924 01:02:22.554366 11148 raft_consensus.cc:2454] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e34471a22eb46589c377044ebe0b485 in term 1.
I20260924 01:02:22.554481 11593 raft_consensus.cc:2781] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:22.557340 11593 raft_consensus.cc:686] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 1 LEADER]: Becoming Leader. State: Replica: 7e34471a22eb46589c377044ebe0b485, State: Running, Role: LEADER
I20260924 01:02:22.557755 11593 consensus_queue.cc:237] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } }
I20260924 01:02:22.561625 10983 catalog_manager.cc:5495] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e34471a22eb46589c377044ebe0b485 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "7e34471a22eb46589c377044ebe0b485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e34471a22eb46589c377044ebe0b485" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 37387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 } health_report { overall_health: UNKNOWN } } }
I20260924 01:02:22.588581 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.582446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.8.254.66:37387)
0924 01:02:22.583221 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.583225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.583243 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:02:22.583337 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.583340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.583411 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.583548 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:22.583927 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.584741 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.586050 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.586543 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.586645 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.587088 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:22.587708 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:22.587824 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.588081 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.588188 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":543,"threads_started":1}
I20260924 01:02:22.604370 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.599147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:39631 (local address 127.8.254.65:34959)
0924 01:02:22.599426 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.599432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.599716 (+ 284us) server_negotiation.cc:407] Connection header received
0924 01:02:22.599878 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.599882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.599948 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.600023 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:22.600575 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.601261 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.602810 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.603298 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.603401 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.603791 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:22.603890 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:22.603957 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.604079 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.604139 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 01:02:22.604430 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.599046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.8.254.66:39631)
0924 01:02:22.599622 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:22.599638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:22.599759 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:22.600059 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:22.600062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:22.600072 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:22.600375 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.600384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.601374 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.601378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.602665 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.602677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.603457 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.603462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.603585 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.603642 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:22.603975 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:22.603984 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 01:02:22.604044 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 01:02:22.604141 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":268,"threads_started":1}
I20260924 01:02:22.612854 11148 raft_consensus.cc:1261] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Refusing update from remote peer 7e34471a22eb46589c377044ebe0b485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:22.612823 11424 raft_consensus.cc:1261] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Refusing update from remote peer 7e34471a22eb46589c377044ebe0b485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:22.613572 11610 consensus_queue.cc:1035] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d938f9bc8f4833b39c157e31199876" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:22.613919 11593 consensus_queue.cc:1035] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e7f53efc428ba677843d65f31fa7" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:22.621253 11557 mvcc.cc:205] Tried to move back new op lower bound from 7332707297736519680 to 7332707297521348608. Current Snapshot: MvccSnapshot[applied={T|T < 7332707297736519680}]
I20260924 01:02:22.641789 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.636137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:50677 (local address 127.8.254.66:37387)
0924 01:02:22.636327 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:22.636334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:22.636845 (+ 511us) server_negotiation.cc:407] Connection header received
0924 01:02:22.636962 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:22.636965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:22.637028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:22.637111 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:22.637560 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.638430 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.640083 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.640509 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.640671 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.641067 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:22.641167 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:22.641269 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:22.641403 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 01:02:22.641481 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:02:22.641881 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:22.636097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.8.254.65:50677)
0924 01:02:22.636746 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:22.636762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:22.636874 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:22.637143 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:22.637146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:22.637155 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:02:22.637390 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.637399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.638588 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.638594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.639933 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:22.639946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:22.640677 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:22.640683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:22.640837 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:22.640876 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:02:22.641274 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:02:22.641286 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 01:02:22.641372 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 01:02:22.641514 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":372,"threads_started":1}
W20260924 01:02:22.643076 11561 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 01:02:24.246505 10985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:24.246800 10985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:24.251178 10985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2a37d0d921e409f93630dab1cbe1619: Sending DeleteTablet for 3 replicas of tablet 769c88db6d8b4bd9ace2b3e6e35d49b1
I20260924 01:02:24.252151 11264 tablet_service.cc:1514] Processing DeleteTablet for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:24 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49760
I20260924 01:02:24.252204 11124 tablet_service.cc:1514] Processing DeleteTablet for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:24 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:46944
I20260924 01:02:24.252226 11402 tablet_service.cc:1514] Processing DeleteTablet for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:24 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49920
I20260924 01:02:24.252946 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11056
I20260924 01:02:24.262701 11645 tablet_replica.cc:330] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: stopping tablet replica
I20260924 01:02:24.263012 11645 raft_consensus.cc:2229] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:24.263358 11645 raft_consensus.cc:2258] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:24.263952 11643 tablet_replica.cc:330] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: stopping tablet replica
I20260924 01:02:24.264207 11643 raft_consensus.cc:2229] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:24.264830 11643 raft_consensus.cc:2258] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:24.265038 11645 ts_tablet_manager.cc:1905] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:24.265400 11644 tablet_replica.cc:330] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: stopping tablet replica
I20260924 01:02:24.265615 11644 raft_consensus.cc:2229] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:24.265879 11644 raft_consensus.cc:2258] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:24.266247 11643 ts_tablet_manager.cc:1905] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:24.268934 11644 ts_tablet_manager.cc:1905] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P baa5e7f53efc428ba677843d65f31fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 01:02:24 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:24.289098 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.255973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.8.254.65:34959)
0924 01:02:24.262529 (+ 6556us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:24.262535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:24.262559 (+ 24us) server_negotiation.cc:407] Connection header received
0924 01:02:24.262626 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:24.262631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:24.262735 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:24.268654 (+ 5919us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:24.268833 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:24.269653 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:24.271025 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:24.271713 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:24.271875 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:24.283028 (+ 11153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:24.283046 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:24.283063 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:24.283105 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:24.286249 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:24.287007 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:24.287014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:24.287017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:24.287175 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:24.287527 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:24.287530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:24.287532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:24.288069 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:24.288382 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:24.288584 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 01:02:24.288813 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6412,"thread_start_us":6245,"threads_started":1}
W20260924 01:02:24.289829 11189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39132
W20260924 01:02:24.290500 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11056: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:24.290747 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 11056
I20260924 01:02:24.334483 11643 ts_tablet_manager.cc:1918] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 01:02:24.334648 11643 log.cc:1196] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/769c88db6d8b4bd9ace2b3e6e35d49b1
I20260924 01:02:24.335124 11643 ts_tablet_manager.cc:1939] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 7e34471a22eb46589c377044ebe0b485: Deleting consensus metadata
I20260924 01:02:24.335579 11645 ts_tablet_manager.cc:1918] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 01:02:24.335750 11645 log.cc:1196] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/769c88db6d8b4bd9ace2b3e6e35d49b1
I20260924 01:02:24.336136 11645 ts_tablet_manager.cc:1939] T 769c88db6d8b4bd9ace2b3e6e35d49b1 P 38d938f9bc8f4833b39c157e31199876: Deleting consensus metadata
W20260924 01:02:24.353250 10972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.254.65:34959 (error 108)
I20260924 01:02:24.354143 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11198
I20260924 01:02:24.355897 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.354826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.0.0.1:39142)
0924 01:02:24.355519 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.355697 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
W20260924 01:02:24.356127 10972 catalog_manager.cc:4584] TS baa5e7f53efc428ba677843d65f31fa7 (127.8.254.65:34959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 769c88db6d8b4bd9ace2b3e6e35d49b1: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34959: connect: Connection refused (error 111)
Sep 24 01:02:24 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:24.371385 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.354947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.8.254.66:37387)
0924 01:02:24.355680 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:24.355685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:24.355700 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:02:24.355756 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:24.355759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:24.355827 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:24.355950 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:24.357380 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:24.358116 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:24.359491 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:24.359994 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:24.360135 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:24.364647 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:24.364664 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:24.364680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:24.364724 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:24.368334 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:24.369207 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:24.369216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:24.369218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:24.369369 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:24.369770 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:24.369773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:24.369775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:24.370164 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:24.370537 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:24.370798 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 01:02:24.370980 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":500,"threads_started":1}
W20260924 01:02:24.372092 11329 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43908
W20260924 01:02:24.372769 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:24.372938 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 11198
I20260924 01:02:24.409864 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.409257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.0.0.1:39152)
0924 01:02:24.409532 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.409655 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 01:02:24.411172 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.410685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.0.0.1:39162)
0924 01:02:24.410877 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.410992 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 01:02:24.422304 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11338
I20260924 01:02:24.422878 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.422421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.0.0.1:43920)
0924 01:02:24.422621 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.422696 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.66:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 01:02:24.424706 10972 catalog_manager.cc:4841] TS 38d938f9bc8f4833b39c157e31199876 (127.8.254.67:46157): tablet 769c88db6d8b4bd9ace2b3e6e35d49b1 (table test-table [id=8644edb4c93644cfbb22ef004be2cd5d]) successfully deleted
Sep 24 01:02:24 dist-test-slave-5c7l krb5kdc[10940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:24.439978 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.423220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.8.254.67:46157)
0924 01:02:24.423871 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:24.423877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:24.423895 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:02:24.423959 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:24.423963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:24.424038 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:24.424150 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:24.425609 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:24.426647 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:24.428255 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:24.428892 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:24.429028 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:24.433749 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:24.433772 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:24.433806 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:24.433862 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:24.437252 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:24.438006 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:24.438014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:24.438016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:24.438164 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:24.438554 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:24.438557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:24.438559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:24.438924 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:24.439147 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:24.439512 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 01:02:24.439682 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":333,"threads_started":1}
W20260924 01:02:24.440766 11469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59032
W20260924 01:02:24.441381 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11338: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:24.441570 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 11338
I20260924 01:02:24.456171 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.455689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.0.0.1:43936)
0924 01:02:24.455911 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.456006 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.66:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 01:02:24.457208 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.456847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.0.0.1:43948)
0924 01:02:24.457021 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.457091 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.66:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 01:02:24.466550 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 10953
I20260924 01:02:24.467792 11025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:24.588393 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.587835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:37387 (local address 127.0.0.1:43958)
0924 01:02:24.588047 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.588195 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.66:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 01:02:24.588881 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 10953
I20260924 01:02:24.588994 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:24.588367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34959 (local address 127.0.0.1:39170)
0924 01:02:24.588686 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:24.588807 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"threads_started":1}
2026-09-24T01:02:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8521 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:26 dist-test-slave-5c7l krb5kdc[11656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:26 dist-test-slave-5c7l krb5kdc[11656](info): set up 2 sockets
Sep 24 01:02:26 dist-test-slave-5c7l krb5kdc[11656](info): commencing operation
krb5kdc: starting...
W20260924 01:02:27.385793 9209 mini_kdc.cc:121] Time spent starting KDC: real 2.750s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 01:02:27.646080 9209 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.260s user 0.003s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 01:02:27.934324 9209 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.288s user 0.002s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 01:02:28.047546 9209 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.113s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:28 dist-test-slave-5c7l krb5kdc[11656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:28.161624 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:35577
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:35577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:28.503952 11669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:28.504328 11669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:28.504489 11669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:28.511778 11669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:28.511942 11669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:28.512055 11669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:28.512162 11669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:28.512238 11669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:28.522248 11669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:35577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:35577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:28.523741 11669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:28.525077 11669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:28.537516 11674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:28.537533 11675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:28.537608 11677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:28.538925 11669 server_base.cc:1034] running on GCE node
I20260924 01:02:28.539831 11669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:28.541426 11669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:28.542666 11669 hybrid_clock.cc:648] HybridClock initialized: now 1790211748542622 us; error 70 us; skew 500 ppm
I20260924 01:02:28.543051 11669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:28 dist-test-slave-5c7l krb5kdc[11656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211748, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:28.546351 11669 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:28.547940 11669 webserver.cc:460] Webserver started at http://127.8.254.126:38269/ using document root <none> and password file <none>
I20260924 01:02:28.548660 11669 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:28.548817 11669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:28.549129 11669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:28.552151 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "764ba66b468c48de87c00d1ecc569b42"
format_stamp: "Formatted at 2026-09-24 01:02:28 on dist-test-slave-5c7l"
server_key: "ecac8ee903d4053a3cdde443ffffdc3e"
server_key_iv: "812380b5157cbf5b24f25684fd8415fd"
server_key_version: "encryptionkey@0"
I20260924 01:02:28.552898 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "764ba66b468c48de87c00d1ecc569b42"
format_stamp: "Formatted at 2026-09-24 01:02:28 on dist-test-slave-5c7l"
server_key: "ecac8ee903d4053a3cdde443ffffdc3e"
server_key_iv: "812380b5157cbf5b24f25684fd8415fd"
server_key_version: "encryptionkey@0"
I20260924 01:02:28.558333 11669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:02:28.561914 11684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:28.563426 11669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 01:02:28.563625 11669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "764ba66b468c48de87c00d1ecc569b42"
format_stamp: "Formatted at 2026-09-24 01:02:28 on dist-test-slave-5c7l"
server_key: "ecac8ee903d4053a3cdde443ffffdc3e"
server_key_iv: "812380b5157cbf5b24f25684fd8415fd"
server_key_version: "encryptionkey@0"
I20260924 01:02:28.563869 11669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:28.581548 11669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:28.586321 11669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:28.586643 11669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:28.605132 11669 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:35577
I20260924 01:02:28.605181 11736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:35577 every 8 connection(s)
I20260924 01:02:28.606642 11669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:28.610316 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 11669
I20260924 01:02:28.610524 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:28.610577 11737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:28.611212 9209 external_mini_cluster.cc:1428] Setting key c686a4c329fe2f1016f7ce69d5d5f614
I20260924 01:02:28.624939 11737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42: Bootstrap starting.
Sep 24 01:02:28 dist-test-slave-5c7l krb5kdc[11656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:28.628857 11737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:28.630288 11737 log.cc:824] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:28.634064 11737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42: No bootstrap required, opened a new log
I20260924 01:02:28.635694 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:28.613201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.8.254.126:35577)
0924 01:02:28.614096 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:28.614113 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:28.614162 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:02:28.615867 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:28.615901 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:28.616445 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:28.616917 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:28.618256 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:28.620122 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:28.621529 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:28.622698 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:28.622894 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:28.626812 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:28.626823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:28.626843 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:28.626892 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:28.630774 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:28.631801 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:28.631812 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:28.631821 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:28.632023 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:28.632403 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:28.632407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:28.632410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:28.633155 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:28.633466 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:28.633816 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 01:02:28.634308 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":338,"threads_started":1}
I20260924 01:02:28.643036 11737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ba66b468c48de87c00d1ecc569b42" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35577 } }
I20260924 01:02:28.643545 11737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:28.643656 11737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764ba66b468c48de87c00d1ecc569b42, State: Initialized, Role: FOLLOWER
I20260924 01:02:28.644311 11737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ba66b468c48de87c00d1ecc569b42" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35577 } }
I20260924 01:02:28.644557 11737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:28.644717 11737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:28.644901 11737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:28.647138 11737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ba66b468c48de87c00d1ecc569b42" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35577 } }
I20260924 01:02:28.647714 11737 leader_election.cc:302] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 764ba66b468c48de87c00d1ecc569b42; no voters:
I20260924 01:02:28.648308 11737 leader_election.cc:288] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:28.648461 11742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:28.649160 11742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [term 1 LEADER]: Becoming Leader. State: Replica: 764ba66b468c48de87c00d1ecc569b42, State: Running, Role: LEADER
I20260924 01:02:28.649677 11742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ba66b468c48de87c00d1ecc569b42" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35577 } }
I20260924 01:02:28.650781 11737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:28.652974 11744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764ba66b468c48de87c00d1ecc569b42. Latest consensus state: current_term: 1 leader_uuid: "764ba66b468c48de87c00d1ecc569b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ba66b468c48de87c00d1ecc569b42" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35577 } } }
I20260924 01:02:28.653299 11743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "764ba66b468c48de87c00d1ecc569b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ba66b468c48de87c00d1ecc569b42" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35577 } } }
I20260924 01:02:28.653527 11744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:28.653599 11743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:28.655699 11750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:28.661396 11750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:28.671535 11750 catalog_manager.cc:1347] Generated new cluster ID: 8886b4dd012c4a618adbb1a78897006c
I20260924 01:02:28.671671 11750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:28.685932 11750 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:28.687721 11750 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:28.699273 11750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 764ba66b468c48de87c00d1ecc569b42: Generated new TSK 0
I20260924 01:02:28.700198 11750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:28.766541 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:35577
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:29.141278 11763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:29.141544 11763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:29.141696 11763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:29.149029 11763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:29.149174 11763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:29.149359 11763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:29.159919 11763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:29.161262 11763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:29.162747 11763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:29.176630 11771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:29.176676 11768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:29.176672 11769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:29.177392 11763 server_base.cc:1034] running on GCE node
I20260924 01:02:29.178069 11763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:29.179102 11763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:29.180322 11763 hybrid_clock.cc:648] HybridClock initialized: now 1790211749180275 us; error 41 us; skew 500 ppm
I20260924 01:02:29.180680 11763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:29 dist-test-slave-5c7l krb5kdc[11656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211749, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:29.184031 11763 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:29.185600 11763 webserver.cc:460] Webserver started at http://127.8.254.65:36439/ using document root <none> and password file <none>
I20260924 01:02:29.186358 11763 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:29.186465 11763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:29.186728 11763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:29.189673 11763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "fa5b7e8162a5494eaea8f131a41293e4"
format_stamp: "Formatted at 2026-09-24 01:02:29 on dist-test-slave-5c7l"
server_key: "9869fc4966ccd8b1bd0abd76cb3ba39a"
server_key_iv: "cf599b58d05326a0de4dca6bd9b4140d"
server_key_version: "encryptionkey@0"
I20260924 01:02:29.190485 11763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "fa5b7e8162a5494eaea8f131a41293e4"
format_stamp: "Formatted at 2026-09-24 01:02:29 on dist-test-slave-5c7l"
server_key: "9869fc4966ccd8b1bd0abd76cb3ba39a"
server_key_iv: "cf599b58d05326a0de4dca6bd9b4140d"
server_key_version: "encryptionkey@0"
I20260924 01:02:29.195713 11763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 01:02:29.199292 11778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:29.200649 11763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 01:02:29.200819 11763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "fa5b7e8162a5494eaea8f131a41293e4"
format_stamp: "Formatted at 2026-09-24 01:02:29 on dist-test-slave-5c7l"
server_key: "9869fc4966ccd8b1bd0abd76cb3ba39a"
server_key_iv: "cf599b58d05326a0de4dca6bd9b4140d"
server_key_version: "encryptionkey@0"
I20260924 01:02:29.201042 11763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:29.219192 11763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:29.223543 11763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:29.223853 11763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:29.224798 11763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:29.226141 11763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:29.226259 11763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:29.226413 11763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:29.226477 11763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:29.259838 11763 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:43981
I20260924 01:02:29.259881 11891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:43981 every 8 connection(s)
I20260924 01:02:29.261142 11763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:29.263047 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 11763
I20260924 01:02:29.263273 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:29.264014 9209 external_mini_cluster.cc:1428] Setting key b243d6634ce6f29b9720975ce11189b0
Sep 24 01:02:29 dist-test-slave-5c7l krb5kdc[11656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211749, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:02:29.282685 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:29.264025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:36703 (local address 127.8.254.126:35577)
0924 01:02:29.264521 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:29.264527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:29.265893 (+ 1366us) server_negotiation.cc:407] Connection header received
0924 01:02:29.267899 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:29.267905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:29.268007 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:29.268124 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:29.269628 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.270419 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.272018 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.272416 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.272522 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:29.276758 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:29.276764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:29.276768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:29.276802 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:29.279515 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:29.280279 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:29.280287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:29.280291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:29.280419 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:29.280848 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:29.280854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:29.280857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:29.281146 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:29.281305 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:29.282036 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 01:02:29.282302 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260924 01:02:29.283814 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:29.264585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:35577 (local address 127.8.254.65:36703)
0924 01:02:29.265741 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:29.265794 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:29.267606 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:29.268372 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:29.268381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:29.268897 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:29.269459 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:29.269475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.270577 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.270582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:29.271866 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:29.271885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.272653 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.272662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:29.273009 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:29.273766 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:29.273800 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:29.276515 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:29.279684 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:29.279694 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:29.279719 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:29.280151 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:29.280533 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:29.280538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:29.280541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:29.280735 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:29.281277 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:29.281285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:29.281646 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 01:02:29.281922 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 01:02:29.282512 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":603,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:02:29.285686 11892 heartbeater.cc:347] Connected to a master server at 127.8.254.126:35577
I20260924 01:02:29.286005 11892 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:29.286860 11892 heartbeater.cc:511] Master 127.8.254.126:35577 requested a full tablet report, sending...
I20260924 01:02:29.289100 11702 ts_manager.cc:194] Registered new tserver with Master: fa5b7e8162a5494eaea8f131a41293e4 (127.8.254.65:43981)
I20260924 01:02:29.291015 11702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:36703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:29.309505 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:35577
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:29.648980 11898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:29.649417 11898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:29.649528 11898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:29.657181 11898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:29.657305 11898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:29.657421 11898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:29.667541 11898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:29.668890 11898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:29.670151 11898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:29.683470 11906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:29.683508 11904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:29.683434 11903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:29.684645 11898 server_base.cc:1034] running on GCE node
I20260924 01:02:29.685367 11898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:29.686393 11898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:29.687636 11898 hybrid_clock.cc:648] HybridClock initialized: now 1790211749687597 us; error 57 us; skew 500 ppm
I20260924 01:02:29.687980 11898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:29 dist-test-slave-5c7l krb5kdc[11656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211749, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:29.695323 11898 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:29.696916 11898 webserver.cc:460] Webserver started at http://127.8.254.66:41325/ using document root <none> and password file <none>
I20260924 01:02:29.697629 11898 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:29.697779 11898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:29.698087 11898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:29.701474 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "b4e76d356d1c4f77846a9edbd8eca931"
format_stamp: "Formatted at 2026-09-24 01:02:29 on dist-test-slave-5c7l"
server_key: "337694424193fcb1f2d02badb2d8a9a5"
server_key_iv: "5d744f01901e6f25c9c9de70f09e02a0"
server_key_version: "encryptionkey@0"
I20260924 01:02:29.702365 11898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "b4e76d356d1c4f77846a9edbd8eca931"
format_stamp: "Formatted at 2026-09-24 01:02:29 on dist-test-slave-5c7l"
server_key: "337694424193fcb1f2d02badb2d8a9a5"
server_key_iv: "5d744f01901e6f25c9c9de70f09e02a0"
server_key_version: "encryptionkey@0"
I20260924 01:02:29.707578 11898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 01:02:29.711030 11913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:29.712455 11898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:29.712785 11898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "b4e76d356d1c4f77846a9edbd8eca931"
format_stamp: "Formatted at 2026-09-24 01:02:29 on dist-test-slave-5c7l"
server_key: "337694424193fcb1f2d02badb2d8a9a5"
server_key_iv: "5d744f01901e6f25c9c9de70f09e02a0"
server_key_version: "encryptionkey@0"
I20260924 01:02:29.713032 11898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:29.748121 11898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:29.752878 11898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:29.753223 11898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:29.754303 11898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:29.755836 11898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:29.755944 11898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:29.756115 11898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:29.756189 11898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:29.791141 11898 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:45611
I20260924 01:02:29.791241 12026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:45611 every 8 connection(s)
I20260924 01:02:29.792543 11898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:29.798830 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 11898
I20260924 01:02:29.799034 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:29.799680 9209 external_mini_cluster.cc:1428] Setting key 195cbe686bb9d69bd8fa018798f2838f
Sep 24 01:02:29 dist-test-slave-5c7l krb5kdc[11656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211749, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:29.817906 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:29.795298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:60095 (local address 127.8.254.126:35577)
0924 01:02:29.795762 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:29.795767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:29.796801 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 01:02:29.798762 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:29.798768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:29.798868 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:29.799004 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:29.801142 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.802138 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.804829 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.805271 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.805367 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:29.810529 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:29.810539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:29.810548 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:29.810596 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:29.814177 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:29.815021 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:29.815030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:29.815036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:29.815201 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:29.815641 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:29.815649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:29.815655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:29.816050 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:29.816209 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:29.817070 (+ 861us) server_negotiation.cc:299] Negotiation successful
0924 01:02:29.817354 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260924 01:02:29.818703 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:29.795922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:35577 (local address 127.8.254.66:60095)
0924 01:02:29.796626 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:29.796671 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:29.798464 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:29.799382 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:29.799396 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:29.800204 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:29.800966 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:29.800987 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.802346 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.802353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:29.804679 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:29.804698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:29.805531 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:29.805542 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:29.805894 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:29.806713 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:29.806754 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:29.810249 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:29.814394 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:29.814401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:29.814416 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:29.814890 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:29.815327 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:29.815330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:29.815333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:29.815515 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:29.816214 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:29.816228 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:29.816674 (+ 446us) client_negotiation.cc:769] Sending connection context
0924 01:02:29.816916 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 01:02:29.817452 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":267,"threads_started":1}
I20260924 01:02:29.820253 12027 heartbeater.cc:347] Connected to a master server at 127.8.254.126:35577
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:02:29.820569 12027 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:29.821293 12027 heartbeater.cc:511] Master 127.8.254.126:35577 requested a full tablet report, sending...
I20260924 01:02:29.822825 11702 ts_manager.cc:194] Registered new tserver with Master: b4e76d356d1c4f77846a9edbd8eca931 (127.8.254.66:45611)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:29.823746 11702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:60095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:29.845938 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:35577
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:30.169482 12033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:30.169776 12033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:30.169868 12033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:30.177209 12033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:30.177335 12033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:30.177450 12033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:30.187680 12033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:30.188958 12033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:30.190281 12033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:30.203557 12038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:30.203598 12041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:30.203625 12039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:30.204555 12033 server_base.cc:1034] running on GCE node
I20260924 01:02:30.205250 12033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:30.206286 12033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:30.207517 12033 hybrid_clock.cc:648] HybridClock initialized: now 1790211750207473 us; error 49 us; skew 500 ppm
I20260924 01:02:30.207854 12033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:30 dist-test-slave-5c7l krb5kdc[11656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211750, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:30.211161 12033 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:30.212733 12033 webserver.cc:460] Webserver started at http://127.8.254.67:44055/ using document root <none> and password file <none>
I20260924 01:02:30.213476 12033 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:30.213632 12033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:30.213948 12033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:30.217168 12033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "970804b35974435eb544524d5813c7b0"
format_stamp: "Formatted at 2026-09-24 01:02:30 on dist-test-slave-5c7l"
server_key: "4078d4c0b919ca8c3818b951e057284f"
server_key_iv: "4b0be412aba331d0c619a4fed9a12298"
server_key_version: "encryptionkey@0"
I20260924 01:02:30.218065 12033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "970804b35974435eb544524d5813c7b0"
format_stamp: "Formatted at 2026-09-24 01:02:30 on dist-test-slave-5c7l"
server_key: "4078d4c0b919ca8c3818b951e057284f"
server_key_iv: "4b0be412aba331d0c619a4fed9a12298"
server_key_version: "encryptionkey@0"
I20260924 01:02:30.223220 12033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 01:02:30.226763 12048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:30.228111 12033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 01:02:30.228318 12033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "970804b35974435eb544524d5813c7b0"
format_stamp: "Formatted at 2026-09-24 01:02:30 on dist-test-slave-5c7l"
server_key: "4078d4c0b919ca8c3818b951e057284f"
server_key_iv: "4b0be412aba331d0c619a4fed9a12298"
server_key_version: "encryptionkey@0"
I20260924 01:02:30.228538 12033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:30.252156 12033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:30.256445 12033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:30.256742 12033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:30.257740 12033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:30.259148 12033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:30.259258 12033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:30.259405 12033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:30.259486 12033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:30.293687 12033 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:43193
I20260924 01:02:30.293735 12161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:43193 every 8 connection(s)
I20260924 01:02:30.294917 11892 heartbeater.cc:503] Master 127.8.254.126:35577 was elected leader, sending a full tablet report...
I20260924 01:02:30.295208 12033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:30.304476 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12033
I20260924 01:02:30.304646 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:30.305217 9209 external_mini_cluster.cc:1428] Setting key 6a52feea9333e0a61232937bca7d0265
Sep 24 01:02:30 dist-test-slave-5c7l krb5kdc[11656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211750, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:30.315469 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:30.298021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:55791 (local address 127.8.254.126:35577)
0924 01:02:30.298236 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:30.298244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:30.299391 (+ 1147us) server_negotiation.cc:407] Connection header received
0924 01:02:30.301188 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:30.301195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:30.301282 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:30.301403 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:30.302915 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:30.303636 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:30.305917 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:30.306496 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:30.306624 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:30.310158 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:30.310163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:30.310166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:30.310243 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:30.312631 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:30.313374 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:30.313378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:30.313380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:30.313494 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:30.313860 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:30.313863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:30.313864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:30.314139 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:30.314315 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:30.315042 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 01:02:30.315232 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:02:30.316532 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:30.298543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:35577 (local address 127.8.254.67:55791)
0924 01:02:30.299233 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:30.299274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:30.300925 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:30.301638 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:30.301646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:30.302144 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:30.302748 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:30.302766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:30.303800 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:30.303808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:30.305766 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:30.305778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:30.306733 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:30.306740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:30.306958 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:30.307724 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:30.307746 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:30.309965 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:30.312777 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:30.312784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:30.312797 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:30.313264 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:30.313587 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:30.313590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:30.313593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:30.313763 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:30.314297 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:30.314303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:30.314619 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 01:02:30.314872 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 01:02:30.315268 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":254,"threads_started":1}
I20260924 01:02:30.318017 12162 heartbeater.cc:347] Connected to a master server at 127.8.254.126:35577
I20260924 01:02:30.318341 12162 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:30.319118 12162 heartbeater.cc:511] Master 127.8.254.126:35577 requested a full tablet report, sending...
I20260924 01:02:30.320822 11701 ts_manager.cc:194] Registered new tserver with Master: 970804b35974435eb544524d5813c7b0 (127.8.254.67:43193)
I20260924 01:02:30.321719 11701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:55791
I20260924 01:02:30.333534 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 01:02:30.365262 12029 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:30.362813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.8.254.126:35577)
0924 01:02:30.362998 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:30.363003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:30.363443 (+ 440us) server_negotiation.cc:407] Connection header received
0924 01:02:30.363609 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:30.363614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:30.363703 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:30.363801 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:30.365000 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39484: BlockingRecv error: recv got EOF from 127.0.0.1:39484 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:02:30.368525 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11763
W20260924 01:02:30.371348 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11763: Not authorized: Client connection negotiation failed: client connection to 127.8.254.65:43981: server requires authentication, but client does not have Kerberos credentials available
I20260924 01:02:30.371565 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 11763
I20260924 01:02:30.387830 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11898
W20260924 01:02:30.390563 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11898: Not authorized: Client connection negotiation failed: client connection to 127.8.254.66:45611: server requires authentication, but client does not have Kerberos credentials available
I20260924 01:02:30.390722 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 11898
I20260924 01:02:30.403606 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12033
W20260924 01:02:30.406234 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12033: Not authorized: Client connection negotiation failed: client connection to 127.8.254.67:43193: server requires authentication, but client does not have Kerberos credentials available
I20260924 01:02:30.406404 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12033
I20260924 01:02:30.421742 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 11669
I20260924 01:02:30.423028 11732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:30.528184 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 11669
2026-09-24T01:02:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5936 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:30 dist-test-slave-5c7l krb5kdc[12179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:30 dist-test-slave-5c7l krb5kdc[12179](info): set up 2 sockets
Sep 24 01:02:30 dist-test-slave-5c7l krb5kdc[12179](info): commencing operation
krb5kdc: starting...
W20260924 01:02:31.667670 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:31 dist-test-slave-5c7l krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:31.882016 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39981
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:32.214169 12192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:32.214586 12192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:32.214749 12192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:32.221868 12192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:32.222043 12192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:32.222154 12192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:32.222291 12192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:32.222375 12192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:32.232160 12192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:32.233814 12192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:32.235330 12192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:32.248353 12198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:32.248592 12197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:32.249343 12192 server_base.cc:1034] running on GCE node
W20260924 01:02:32.249397 12200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:32.250382 12192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:32.251893 12192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:32.253134 12192 hybrid_clock.cc:648] HybridClock initialized: now 1790211752253082 us; error 90 us; skew 500 ppm
I20260924 01:02:32.253530 12192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:32 dist-test-slave-5c7l krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:32.257201 12192 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:32.258687 12192 webserver.cc:460] Webserver started at http://127.8.254.126:39773/ using document root <none> and password file <none>
I20260924 01:02:32.259353 12192 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:32.259466 12192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:32.259763 12192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:32.262866 12192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "c0860aeb98584964a7ffd51def581d8c"
format_stamp: "Formatted at 2026-09-24 01:02:32 on dist-test-slave-5c7l"
server_key: "141d18d4ba958e533640ec8137392e0e"
server_key_iv: "0811d594b62dc050d4644911613977c8"
server_key_version: "encryptionkey@0"
I20260924 01:02:32.263594 12192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "c0860aeb98584964a7ffd51def581d8c"
format_stamp: "Formatted at 2026-09-24 01:02:32 on dist-test-slave-5c7l"
server_key: "141d18d4ba958e533640ec8137392e0e"
server_key_iv: "0811d594b62dc050d4644911613977c8"
server_key_version: "encryptionkey@0"
I20260924 01:02:32.268620 12192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:32.272145 12207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:32.273531 12192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:32.273676 12192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "c0860aeb98584964a7ffd51def581d8c"
format_stamp: "Formatted at 2026-09-24 01:02:32 on dist-test-slave-5c7l"
server_key: "141d18d4ba958e533640ec8137392e0e"
server_key_iv: "0811d594b62dc050d4644911613977c8"
server_key_version: "encryptionkey@0"
I20260924 01:02:32.273857 12192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:32.292470 12192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:32.296669 12192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:32.296926 12192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:32.316111 12192 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39981
I20260924 01:02:32.316119 12259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39981 every 8 connection(s)
I20260924 01:02:32.317687 12192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:32.321267 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12192
I20260924 01:02:32.321523 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:32.321617 12260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:32.322252 9209 external_mini_cluster.cc:1428] Setting key 3e3732fe90bfa4791c6ac6ab1d130424
I20260924 01:02:32.336544 12260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c: Bootstrap starting.
Sep 24 01:02:32 dist-test-slave-5c7l krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:32.340584 12260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:32.341969 12260 log.cc:824] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:32.345454 12260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c: No bootstrap required, opened a new log
I20260924 01:02:32.346907 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:32.324187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.8.254.126:39981)
0924 01:02:32.325053 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:32.325065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:32.325105 (+ 40us) server_negotiation.cc:407] Connection header received
0924 01:02:32.326690 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:32.326711 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:32.327119 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:32.327459 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:32.329352 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:32.330679 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:32.332441 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:32.333147 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:32.333274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:32.338067 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:32.338084 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:32.338106 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:32.338169 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:32.342139 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:32.343049 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:32.343056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:32.343063 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:32.343249 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:32.343655 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:32.343663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:32.343668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:32.344368 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:32.344660 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:32.345048 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 01:02:32.345502 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":421,"threads_started":1}
I20260924 01:02:32.351038 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:02:32.353891 12260 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0860aeb98584964a7ffd51def581d8c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39981 } }
I20260924 01:02:32.354317 12260 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:32.354422 12260 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0860aeb98584964a7ffd51def581d8c, State: Initialized, Role: FOLLOWER
I20260924 01:02:32.355151 12260 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0860aeb98584964a7ffd51def581d8c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39981 } }
I20260924 01:02:32.355394 12260 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:32.355533 12260 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:32.355703 12260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:32.357896 12260 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0860aeb98584964a7ffd51def581d8c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39981 } }
I20260924 01:02:32.358551 12260 leader_election.cc:302] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0860aeb98584964a7ffd51def581d8c; no voters:
I20260924 01:02:32.359215 12260 leader_election.cc:288] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:32.359323 12266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:32.361877 12260 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:32.365751 12266 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [term 1 LEADER]: Becoming Leader. State: Replica: c0860aeb98584964a7ffd51def581d8c, State: Running, Role: LEADER
I20260924 01:02:32.366734 12266 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0860aeb98584964a7ffd51def581d8c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39981 } }
I20260924 01:02:32.370625 12267 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0860aeb98584964a7ffd51def581d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0860aeb98584964a7ffd51def581d8c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39981 } } }
I20260924 01:02:32.371006 12267 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:32.371778 12268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0860aeb98584964a7ffd51def581d8c. Latest consensus state: current_term: 1 leader_uuid: "c0860aeb98584964a7ffd51def581d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0860aeb98584964a7ffd51def581d8c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39981 } } }
I20260924 01:02:32.372120 12268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c [sys.catalog]: This master's current role is: LEADER
W20260924 01:02:32.377696 12280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:02:32.377826 12280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:02:32.377960 12282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:32.380149 12282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:32.391014 12282 catalog_manager.cc:1347] Generated new cluster ID: 7f7773e8a4204e6a85d9a1c8c9b85db5
I20260924 01:02:32.391136 12282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:32.399881 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:32.390063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.8.254.126:39981)
0924 01:02:32.390281 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:32.390295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:32.391458 (+ 1163us) server_negotiation.cc:407] Connection header received
0924 01:02:32.391652 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:32.391658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:32.391756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:32.391889 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:32.393160 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:32.394374 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:32.396124 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:32.396833 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:32.396967 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:32.398134 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:32.398142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:32.398147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:32.399055 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:32.399123 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:32.399158 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:32.399271 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:32.399413 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 01:02:32.399519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:02:32.404870 12282 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:32.406179 12282 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:32.413302 12282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0860aeb98584964a7ffd51def581d8c: Generated new TSK 0
I20260924 01:02:32.414108 12282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:02:32.428824 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12192
I20260924 01:02:32.430124 12255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:32.521524 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12192
2026-09-24T01:02:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1986 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:32 dist-test-slave-5c7l krb5kdc[12294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:32 dist-test-slave-5c7l krb5kdc[12294](info): set up 2 sockets
Sep 24 01:02:32 dist-test-slave-5c7l krb5kdc[12294](info): commencing operation
krb5kdc: starting...
W20260924 01:02:33.603700 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:33 dist-test-slave-5c7l krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:33.803246 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:41651
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:41651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:34.127238 12307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:34.127542 12307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:34.127733 12307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:34.135087 12307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:34.135211 12307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:34.135326 12307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:34.135423 12307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:34.135501 12307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:34.145462 12307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:41651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:41651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:34.146833 12307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:34.148109 12307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:34.160270 12312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:34.160362 12315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:34.160362 12313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:34.161387 12307 server_base.cc:1034] running on GCE node
I20260924 01:02:34.162282 12307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:34.163763 12307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:34.165009 12307 hybrid_clock.cc:648] HybridClock initialized: now 1790211754164969 us; error 46 us; skew 500 ppm
I20260924 01:02:34.165387 12307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:34 dist-test-slave-5c7l krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211754, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:34.168712 12307 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:34.170238 12307 webserver.cc:460] Webserver started at http://127.8.254.126:46779/ using document root <none> and password file <none>
I20260924 01:02:34.170984 12307 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:34.171120 12307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:34.171422 12307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:34.174532 12307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "0af68ab68cf64375b828148cd7a79277"
format_stamp: "Formatted at 2026-09-24 01:02:34 on dist-test-slave-5c7l"
server_key: "1ec75b231a1564f39f3a41c486b08f82"
server_key_iv: "1a0d6bca8f00122878279fa5b6aeb1e3"
server_key_version: "encryptionkey@0"
I20260924 01:02:34.175264 12307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "0af68ab68cf64375b828148cd7a79277"
format_stamp: "Formatted at 2026-09-24 01:02:34 on dist-test-slave-5c7l"
server_key: "1ec75b231a1564f39f3a41c486b08f82"
server_key_iv: "1a0d6bca8f00122878279fa5b6aeb1e3"
server_key_version: "encryptionkey@0"
I20260924 01:02:34.180923 12307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:34.184478 12322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:34.185824 12307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:02:34.185971 12307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "0af68ab68cf64375b828148cd7a79277"
format_stamp: "Formatted at 2026-09-24 01:02:34 on dist-test-slave-5c7l"
server_key: "1ec75b231a1564f39f3a41c486b08f82"
server_key_iv: "1a0d6bca8f00122878279fa5b6aeb1e3"
server_key_version: "encryptionkey@0"
I20260924 01:02:34.186138 12307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:34.202353 12307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:34.206885 12307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:34.207129 12307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:34.225066 12307 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:41651
I20260924 01:02:34.225090 12374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:41651 every 8 connection(s)
I20260924 01:02:34.226531 12307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:34.230391 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:34.232380 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12307
I20260924 01:02:34.232617 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:34.233271 9209 external_mini_cluster.cc:1428] Setting key 34ed7109303f4ed9b5106beeac9aa5a8
I20260924 01:02:34.243484 12375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277: Bootstrap starting.
I20260924 01:02:34.247023 12375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:34.248000 12375 log.cc:824] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277: Log is configured to *not* fsync() on all Append() calls
Sep 24 01:02:34 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:34.251706 12375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277: No bootstrap required, opened a new log
I20260924 01:02:34.258539 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:34.235355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.8.254.126:41651)
0924 01:02:34.236203 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:34.236219 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:34.236268 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:02:34.238625 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:34.238653 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:34.239169 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:34.239624 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:34.240866 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:34.242853 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:34.244111 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:34.245170 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:34.245368 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:34.249620 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:34.249630 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:34.249649 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:34.249704 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:34.253693 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:34.254483 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:34.254496 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:34.254510 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:34.254748 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:34.255113 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:34.255119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:34.255122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:34.255896 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:34.256213 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:34.256560 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 01:02:34.257069 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":343,"threads_started":1}
I20260924 01:02:34.262097 12375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af68ab68cf64375b828148cd7a79277" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41651 } }
I20260924 01:02:34.262524 12375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:34.262603 12375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af68ab68cf64375b828148cd7a79277, State: Initialized, Role: FOLLOWER
I20260924 01:02:34.263168 12375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af68ab68cf64375b828148cd7a79277" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41651 } }
I20260924 01:02:34.263372 12375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:34.263468 12375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:34.263586 12375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:34.265969 12375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af68ab68cf64375b828148cd7a79277" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41651 } }
I20260924 01:02:34.266484 12375 leader_election.cc:302] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af68ab68cf64375b828148cd7a79277; no voters:
I20260924 01:02:34.267092 12375 leader_election.cc:288] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:34.267181 12380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:34.267913 12380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [term 1 LEADER]: Becoming Leader. State: Replica: 0af68ab68cf64375b828148cd7a79277, State: Running, Role: LEADER
I20260924 01:02:34.268577 12380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af68ab68cf64375b828148cd7a79277" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41651 } }
I20260924 01:02:34.269383 12375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:34.271574 12382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af68ab68cf64375b828148cd7a79277. Latest consensus state: current_term: 1 leader_uuid: "0af68ab68cf64375b828148cd7a79277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af68ab68cf64375b828148cd7a79277" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41651 } } }
I20260924 01:02:34.271999 12382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:34.274461 12381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af68ab68cf64375b828148cd7a79277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af68ab68cf64375b828148cd7a79277" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41651 } } }
I20260924 01:02:34.274801 12381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:34.274886 12386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:34.279836 12386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:34.289624 12386 catalog_manager.cc:1347] Generated new cluster ID: 1852301b321c4c72b32513cbe52138e0
I20260924 01:02:34.289783 12386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:34.326468 12386 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:34.328486 12386 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:34.338771 12386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af68ab68cf64375b828148cd7a79277: Generated new TSK 0
I20260924 01:02:34.339897 12386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:34.398983 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41651
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:34.726321 12401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:34.726706 12401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:34.726893 12401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:34.734521 12401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:34.734704 12401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:34.734898 12401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:34.745714 12401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:34.747159 12401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:34.748624 12401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:34.762559 12406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:34.762621 12409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:34.762600 12407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:34.763322 12401 server_base.cc:1034] running on GCE node
I20260924 01:02:34.763975 12401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:34.764974 12401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:34.766285 12401 hybrid_clock.cc:648] HybridClock initialized: now 1790211754766162 us; error 50 us; skew 500 ppm
I20260924 01:02:34.766637 12401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:34 dist-test-slave-5c7l krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211754, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:34.776945 12401 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:34.778748 12401 webserver.cc:460] Webserver started at http://127.8.254.65:39717/ using document root <none> and password file <none>
I20260924 01:02:34.779537 12401 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:34.779677 12401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:34.779997 12401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:34.783054 12401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "d139bd6f7d064409bec4070e56f11bed"
format_stamp: "Formatted at 2026-09-24 01:02:34 on dist-test-slave-5c7l"
server_key: "bb6e1e4a29f9b4bd7e2fdcbdb3680c7e"
server_key_iv: "eaf71cc29954ef35b535c7a39d1e604a"
server_key_version: "encryptionkey@0"
I20260924 01:02:34.783881 12401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "d139bd6f7d064409bec4070e56f11bed"
format_stamp: "Formatted at 2026-09-24 01:02:34 on dist-test-slave-5c7l"
server_key: "bb6e1e4a29f9b4bd7e2fdcbdb3680c7e"
server_key_iv: "eaf71cc29954ef35b535c7a39d1e604a"
server_key_version: "encryptionkey@0"
I20260924 01:02:34.789345 12401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:34.793359 12416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:34.794935 12401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:34.795122 12401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "d139bd6f7d064409bec4070e56f11bed"
format_stamp: "Formatted at 2026-09-24 01:02:34 on dist-test-slave-5c7l"
server_key: "bb6e1e4a29f9b4bd7e2fdcbdb3680c7e"
server_key_iv: "eaf71cc29954ef35b535c7a39d1e604a"
server_key_version: "encryptionkey@0"
I20260924 01:02:34.795342 12401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:34.826548 12401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:34.831018 12401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:34.831347 12401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:34.832366 12401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:34.833835 12401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:34.833913 12401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:34.834043 12401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:34.834113 12401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:34.869714 12401 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:46087
I20260924 01:02:34.869822 12529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:46087 every 8 connection(s)
I20260924 01:02:34.871276 12401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:34.880089 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12401
I20260924 01:02:34.880321 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:34.880949 9209 external_mini_cluster.cc:1428] Setting key 9144346003d39e975405f69799422654
Sep 24 01:02:34 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211754, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:34.892088 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:34.873750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:49419 (local address 127.8.254.126:41651)
0924 01:02:34.874234 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:34.874240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:34.875237 (+ 997us) server_negotiation.cc:407] Connection header received
0924 01:02:34.877026 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:34.877030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:34.877101 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:34.877207 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:34.878825 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:34.879603 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:34.881377 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:34.881859 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:34.881967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:34.886012 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:34.886017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:34.886021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:34.886051 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:34.888732 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:34.889584 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:34.889589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:34.889592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:34.889709 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:34.890214 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:34.890219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:34.890222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:34.890493 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:34.890653 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:34.891484 (+ 831us) server_negotiation.cc:299] Negotiation successful
0924 01:02:34.891714 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":207,"threads_started":1}
I20260924 01:02:34.893077 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:34.874369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41651 (local address 127.8.254.65:49419)
0924 01:02:34.875072 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:34.875111 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:34.876747 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:34.877420 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:34.877427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:34.877932 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:34.878601 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:34.878619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:34.879774 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:34.879778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:34.881225 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:34.881238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:34.882087 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:34.882097 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:34.882378 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:34.883360 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:34.883383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:34.885791 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:34.888915 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:34.888924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:34.888949 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:34.889439 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:34.889822 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:34.889828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:34.889834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:34.890045 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:34.890636 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:34.890644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:34.890991 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 01:02:34.891364 (+ 373us) client_negotiation.cc:240] Negotiation successful
0924 01:02:34.891936 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":268,"threads_started":1}
I20260924 01:02:34.894783 12530 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41651
I20260924 01:02:34.895135 12530 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:34.895931 12530 heartbeater.cc:511] Master 127.8.254.126:41651 requested a full tablet report, sending...
I20260924 01:02:34.898314 12339 ts_manager.cc:194] Registered new tserver with Master: d139bd6f7d064409bec4070e56f11bed (127.8.254.65:46087)
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:02:34.899971 12339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:49419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:34.927114 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41651
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:35.258967 12536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:35.259264 12536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:35.259418 12536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:35.267016 12536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:35.267132 12536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:35.267300 12536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:35.277453 12536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:35.278910 12536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:35.280207 12536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:35.293458 12544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:35.293496 12542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:35.293393 12541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:35.294461 12536 server_base.cc:1034] running on GCE node
I20260924 01:02:35.295228 12536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:35.296217 12536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:35.297449 12536 hybrid_clock.cc:648] HybridClock initialized: now 1790211755297393 us; error 53 us; skew 500 ppm
I20260924 01:02:35.297796 12536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:35.301323 12536 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:35.302902 12536 webserver.cc:460] Webserver started at http://127.8.254.66:36343/ using document root <none> and password file <none>
I20260924 01:02:35.303639 12536 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:35.303804 12536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:35.304137 12536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:35.307420 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "b5fa1e39106f4d8d9728037c10ace0ac"
format_stamp: "Formatted at 2026-09-24 01:02:35 on dist-test-slave-5c7l"
server_key: "a86e8499af812855ffd6176505d2cab6"
server_key_iv: "18aef35842a5f13de61a6bfefc7dc0b2"
server_key_version: "encryptionkey@0"
I20260924 01:02:35.308301 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "b5fa1e39106f4d8d9728037c10ace0ac"
format_stamp: "Formatted at 2026-09-24 01:02:35 on dist-test-slave-5c7l"
server_key: "a86e8499af812855ffd6176505d2cab6"
server_key_iv: "18aef35842a5f13de61a6bfefc7dc0b2"
server_key_version: "encryptionkey@0"
I20260924 01:02:35.313848 12536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:35.317543 12551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:35.319020 12536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:35.319216 12536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "b5fa1e39106f4d8d9728037c10ace0ac"
format_stamp: "Formatted at 2026-09-24 01:02:35 on dist-test-slave-5c7l"
server_key: "a86e8499af812855ffd6176505d2cab6"
server_key_iv: "18aef35842a5f13de61a6bfefc7dc0b2"
server_key_version: "encryptionkey@0"
I20260924 01:02:35.319453 12536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:35.347450 12536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:35.351969 12536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:35.352340 12536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:35.353504 12536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:35.355010 12536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:35.355129 12536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:35.355300 12536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:35.355388 12536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:35.392628 12536 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:36883
I20260924 01:02:35.392647 12664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:36883 every 8 connection(s)
I20260924 01:02:35.394160 12536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:35.397084 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12536
I20260924 01:02:35.397343 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:35.397867 9209 external_mini_cluster.cc:1428] Setting key 8244aeb385ab027fd5fc3d4f2ff8e09c
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:02:35.418831 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:35.396900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:40629 (local address 127.8.254.126:41651)
0924 01:02:35.397412 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:35.397418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:35.398631 (+ 1213us) server_negotiation.cc:407] Connection header received
0924 01:02:35.401265 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:35.401269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:35.401333 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:35.401436 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:35.403795 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.404506 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.407051 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.407530 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.407688 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:35.411570 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:35.411579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:35.411587 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:35.411633 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:35.415446 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.416196 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.416203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.416207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.416388 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.416805 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.416812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.416817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.417205 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:35.417379 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:35.418071 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 01:02:35.418363 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260924 01:02:35.419536 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:35.397393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41651 (local address 127.8.254.66:40629)
0924 01:02:35.398312 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:35.398369 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:35.400946 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:35.401762 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:35.401778 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:35.402694 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:35.403600 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:35.403632 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.404679 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.404689 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:35.406879 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:35.406898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.407794 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.407802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:35.408058 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:35.408819 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:35.408843 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:35.411302 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:35.415592 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:35.415599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:35.415612 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:35.416054 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:35.416500 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:35.416503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:35.416505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:35.416678 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:35.417326 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:35.417332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:35.417642 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 01:02:35.417888 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 01:02:35.418371 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":319,"threads_started":1}
I20260924 01:02:35.421201 12665 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41651
I20260924 01:02:35.421499 12665 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:35.422497 12665 heartbeater.cc:511] Master 127.8.254.126:41651 requested a full tablet report, sending...
I20260924 01:02:35.424226 12339 ts_manager.cc:194] Registered new tserver with Master: b5fa1e39106f4d8d9728037c10ace0ac (127.8.254.66:36883)
I20260924 01:02:35.425161 12339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:40629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:35.446475 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41651
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:35.773701 12671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:35.773996 12671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:35.774152 12671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:35.781744 12671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:35.781863 12671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:35.782063 12671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:35.792095 12671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:35.793505 12671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:35.794935 12671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:35.808341 12676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:35.808413 12677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:35.808563 12679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:35.809046 12671 server_base.cc:1034] running on GCE node
I20260924 01:02:35.809680 12671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:35.810680 12671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:35.811910 12671 hybrid_clock.cc:648] HybridClock initialized: now 1790211755811850 us; error 52 us; skew 500 ppm
I20260924 01:02:35.812218 12671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:35.815383 12671 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:35.816926 12671 webserver.cc:460] Webserver started at http://127.8.254.67:33259/ using document root <none> and password file <none>
I20260924 01:02:35.817621 12671 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:35.817770 12671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:35.818075 12671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:35.821000 12671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "0869a9848bac4165bf653c6ae56d53ad"
format_stamp: "Formatted at 2026-09-24 01:02:35 on dist-test-slave-5c7l"
server_key: "11519c4b579d78a01955c890f2bcca0f"
server_key_iv: "53036e2e8d3dc6ac958093ad2e212f35"
server_key_version: "encryptionkey@0"
I20260924 01:02:35.821774 12671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "0869a9848bac4165bf653c6ae56d53ad"
format_stamp: "Formatted at 2026-09-24 01:02:35 on dist-test-slave-5c7l"
server_key: "11519c4b579d78a01955c890f2bcca0f"
server_key_iv: "53036e2e8d3dc6ac958093ad2e212f35"
server_key_version: "encryptionkey@0"
I20260924 01:02:35.827136 12671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:02:35.830616 12686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:35.831962 12671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 01:02:35.832142 12671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "0869a9848bac4165bf653c6ae56d53ad"
format_stamp: "Formatted at 2026-09-24 01:02:35 on dist-test-slave-5c7l"
server_key: "11519c4b579d78a01955c890f2bcca0f"
server_key_iv: "53036e2e8d3dc6ac958093ad2e212f35"
server_key_version: "encryptionkey@0"
I20260924 01:02:35.832363 12671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:35.847014 12671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:35.850848 12671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:35.851146 12671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:35.852103 12671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:35.853413 12671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:35.853518 12671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:35.853667 12671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:35.853744 12671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:35.886471 12671 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:34999
I20260924 01:02:35.886504 12799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:34999 every 8 connection(s)
I20260924 01:02:35.887805 12671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:35.894832 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12671
I20260924 01:02:35.895037 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:35.895556 9209 external_mini_cluster.cc:1428] Setting key 3b7bb6617db7528a337fe2bad896e025
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:35.903347 12530 heartbeater.cc:503] Master 127.8.254.126:41651 was elected leader, sending a full tablet report...
I20260924 01:02:35.907944 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:35.890438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:36055 (local address 127.8.254.126:41651)
0924 01:02:35.890654 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:35.890660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:35.891861 (+ 1201us) server_negotiation.cc:407] Connection header received
0924 01:02:35.893603 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:35.893608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:35.893679 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:35.893768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:35.895411 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.896439 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.898253 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.898745 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.898894 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:35.902307 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:35.902314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:35.902319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:35.902362 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:35.905336 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.906007 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.906010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.906012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.906122 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.906540 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.906543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.906545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.906811 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:35.906934 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:35.907557 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 01:02:35.907728 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:02:35.908983 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:35.891002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41651 (local address 127.8.254.67:36055)
0924 01:02:35.891704 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:35.891740 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:35.893332 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:35.893983 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:35.893990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:35.894559 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:35.895234 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:35.895253 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.896615 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.896621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:35.898059 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:35.898071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.898973 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.898980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:35.899220 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:35.899858 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:35.899882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:35.902067 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:35.905462 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:35.905468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:35.905480 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:35.905907 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:35.906255 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:35.906260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:35.906263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:35.906445 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:35.906925 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:35.906931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:35.907247 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 01:02:35.907462 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 01:02:35.907831 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260924 01:02:35.910512 12800 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41651
I20260924 01:02:35.910841 12800 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:35.911631 12800 heartbeater.cc:511] Master 127.8.254.126:41651 requested a full tablet report, sending...
I20260924 01:02:35.912988 12338 ts_manager.cc:194] Registered new tserver with Master: 0869a9848bac4165bf653c6ae56d53ad (127.8.254.67:34999)
I20260924 01:02:35.913894 12338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:36055
I20260924 01:02:35.923611 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:35.972216 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:35.957914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.8.254.126:41651)
0924 01:02:35.958094 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:35.958098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:35.958692 (+ 594us) server_negotiation.cc:407] Connection header received
0924 01:02:35.958930 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:35.958935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:35.959023 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:35.959133 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:35.960495 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.961231 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.962646 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.963049 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.963189 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:35.966982 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:35.966987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:35.966991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:35.967024 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:35.969471 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.970124 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.970133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.970136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.970363 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.970747 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.970752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.970755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.971082 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:35.971351 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:35.971640 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 01:02:35.971860 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 01:02:35.972795 12338 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37838
I20260924 01:02:35.974493 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12401
Sep 24 01:02:35 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:35.990808 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:35.975610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.8.254.65:46087)
0924 01:02:35.976250 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:35.976257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:35.976280 (+ 23us) server_negotiation.cc:407] Connection header received
0924 01:02:35.976335 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:35.976341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:35.976552 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:35.976726 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:35.977964 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.979100 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.980378 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:35.980924 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:35.981058 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:35.984952 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:35.984963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:35.984979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:35.985017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:35.987841 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.988568 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.988575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.988578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.988703 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:35.989088 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:35.989093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:35.989096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:35.989504 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:35.989811 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:35.990121 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 01:02:35.990450 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":328,"threads_started":1}
W20260924 01:02:35.992107 12525 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46426
W20260924 01:02:35.993328 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12401: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:35.993536 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12401
I20260924 01:02:36.010005 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12536
Sep 24 01:02:36 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:36.025933 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:36.011013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.8.254.66:36883)
0924 01:02:36.011611 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:36.011618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:36.011646 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:02:36.011708 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:36.011714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:36.011936 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:36.012064 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:36.013257 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:36.014357 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:36.015551 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:36.016096 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:36.016199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:36.020205 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:36.020216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:36.020232 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:36.020266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:36.023320 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:36.023988 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:36.023995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:36.023998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:36.024125 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:36.024464 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:36.024467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:36.024469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:36.024819 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:36.025050 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:36.025307 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 01:02:36.025507 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":284,"threads_started":1}
W20260924 01:02:36.027374 12659 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55204
W20260924 01:02:36.028482 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:36.028671 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12536
I20260924 01:02:36.044777 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12671
Sep 24 01:02:36 dist-test-slave-5c7l krb5kdc[12294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211755, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:36.060943 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:36.045749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.8.254.67:34999)
0924 01:02:36.046442 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:36.046449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:36.046477 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:02:36.046539 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:36.046545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:36.046769 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:36.046959 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:36.048290 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:36.049362 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:36.050685 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:36.051251 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:36.051388 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:36.055655 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:36.055667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:36.055682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:36.055717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:36.058562 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:36.059171 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:36.059179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:36.059181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:36.059309 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:36.059624 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:36.059627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:36.059629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:36.059973 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:36.060213 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:36.060446 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 01:02:36.060643 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":334,"threads_started":1}
W20260924 01:02:36.062297 12794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35672
W20260924 01:02:36.063472 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:02:36.063628 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12671
I20260924 01:02:36.079825 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12307
I20260924 01:02:36.080976 12370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:36.184558 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12307
2026-09-24T01:02:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3668 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:36 dist-test-slave-5c7l krb5kdc[12817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:36 dist-test-slave-5c7l krb5kdc[12817](info): set up 2 sockets
Sep 24 01:02:36 dist-test-slave-5c7l krb5kdc[12817](info): commencing operation
krb5kdc: starting...
W20260924 01:02:37.271591 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:37 dist-test-slave-5c7l krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:37.469846 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39625
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:37.793789 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:37.794090 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:37.794270 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:37.802503 12830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:37.802613 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:37.802713 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:37.802814 12830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:37.802897 12830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:37.813163 12830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:37.814579 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:37.815907 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:37.827584 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:37.827673 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:37.827630 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:37.828354 12830 server_base.cc:1034] running on GCE node
I20260924 01:02:37.829237 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:37.830742 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:37.831977 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790211757831942 us; error 57 us; skew 500 ppm
I20260924 01:02:37.832338 12830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:37 dist-test-slave-5c7l krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211757, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:37.835457 12830 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:37.836999 12830 webserver.cc:460] Webserver started at http://127.8.254.126:38769/ using document root <none> and password file <none>
I20260924 01:02:37.837692 12830 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:37.837842 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:37.838160 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:37.841255 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "ad177b5ddf0d402cb4f0475f086f6e1d"
format_stamp: "Formatted at 2026-09-24 01:02:37 on dist-test-slave-5c7l"
server_key: "f218389d1fa11bdfa1d571dcbca5cea0"
server_key_iv: "a1ea7118962b76e70757059a9b9bf497"
server_key_version: "encryptionkey@0"
I20260924 01:02:37.842028 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "ad177b5ddf0d402cb4f0475f086f6e1d"
format_stamp: "Formatted at 2026-09-24 01:02:37 on dist-test-slave-5c7l"
server_key: "f218389d1fa11bdfa1d571dcbca5cea0"
server_key_iv: "a1ea7118962b76e70757059a9b9bf497"
server_key_version: "encryptionkey@0"
I20260924 01:02:37.847267 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 01:02:37.850608 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:37.851873 12830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 01:02:37.852057 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "ad177b5ddf0d402cb4f0475f086f6e1d"
format_stamp: "Formatted at 2026-09-24 01:02:37 on dist-test-slave-5c7l"
server_key: "f218389d1fa11bdfa1d571dcbca5cea0"
server_key_iv: "a1ea7118962b76e70757059a9b9bf497"
server_key_version: "encryptionkey@0"
I20260924 01:02:37.852268 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:37.864667 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:37.868767 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:37.869048 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:37.887522 12830 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39625
I20260924 01:02:37.887545 12896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39625 every 8 connection(s)
I20260924 01:02:37.888926 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:37.890021 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12830
I20260924 01:02:37.890265 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:37.891073 9209 external_mini_cluster.cc:1428] Setting key d83212b7358b31f58bff5bf6968fe48a
I20260924 01:02:37.894014 12897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 01:02:37 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:37.907056 12897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d: Bootstrap starting.
I20260924 01:02:37.911315 12897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:37.912565 12897 log.cc:824] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:37.915750 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:37.893142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.8.254.126:39625)
0924 01:02:37.893914 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:37.893931 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:37.893984 (+ 53us) server_negotiation.cc:407] Connection header received
0924 01:02:37.896314 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:37.896342 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:37.896827 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:37.897299 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:37.898658 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:37.900749 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:37.901959 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:37.902992 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:37.903173 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:37.907186 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:37.907198 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:37.907218 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:37.907259 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:37.911142 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:37.911875 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:37.911883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:37.911890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:37.912091 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:37.912489 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:37.912494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:37.912497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:37.913239 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:37.913490 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:37.913988 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 01:02:37.914456 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":338,"threads_started":1}
I20260924 01:02:37.916079 12897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d: No bootstrap required, opened a new log
I20260924 01:02:37.923748 12897 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39625 } }
I20260924 01:02:37.924144 12897 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:37.924268 12897 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad177b5ddf0d402cb4f0475f086f6e1d, State: Initialized, Role: FOLLOWER
I20260924 01:02:37.924942 12897 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39625 } }
I20260924 01:02:37.925259 12897 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:37.925395 12897 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:37.925577 12897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:37.927793 12897 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39625 } }
I20260924 01:02:37.928309 12897 leader_election.cc:302] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad177b5ddf0d402cb4f0475f086f6e1d; no voters:
I20260924 01:02:37.928890 12897 leader_election.cc:288] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:37.929159 12902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:37.930183 12902 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [term 1 LEADER]: Becoming Leader. State: Replica: ad177b5ddf0d402cb4f0475f086f6e1d, State: Running, Role: LEADER
I20260924 01:02:37.930889 12902 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39625 } }
I20260924 01:02:37.931239 12897 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:37.933719 12904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad177b5ddf0d402cb4f0475f086f6e1d. Latest consensus state: current_term: 1 leader_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39625 } } }
I20260924 01:02:37.934183 12904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:37.934393 12903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad177b5ddf0d402cb4f0475f086f6e1d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39625 } } }
I20260924 01:02:37.934755 12903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d [sys.catalog]: This master's current role is: LEADER
W20260924 01:02:37.943881 12918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:02:37.944036 12918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:02:37.944176 12911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:37.945820 12911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:37.953958 12911 catalog_manager.cc:1347] Generated new cluster ID: 1267773f9b604235907d8f3c70a43393
I20260924 01:02:37.954087 12911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:37.969343 12911 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:37.970781 12911 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:37.976577 12911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad177b5ddf0d402cb4f0475f086f6e1d: Generated new TSK 0
I20260924 01:02:37.977432 12911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:38.030861 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39625
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:38.355566 12923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:38.355904 12923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:38.356071 12923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:38.363643 12923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:38.363759 12923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:38.363871 12923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:38.374274 12923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:38.375550 12923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:38.376787 12923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:38.390018 12928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:38.390012 12929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:38.390380 12931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:38.391469 12923 server_base.cc:1034] running on GCE node
I20260924 01:02:38.392088 12923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:38.392994 12923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:38.394289 12923 hybrid_clock.cc:648] HybridClock initialized: now 1790211758394244 us; error 60 us; skew 500 ppm
I20260924 01:02:38.394606 12923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:38 dist-test-slave-5c7l krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:38.397710 12923 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:38.399195 12923 webserver.cc:460] Webserver started at http://127.8.254.65:33197/ using document root <none> and password file <none>
I20260924 01:02:38.399865 12923 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:38.400014 12923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:38.400323 12923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:38.403246 12923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "103bef62b67e4d409ea1b8a43e868133"
format_stamp: "Formatted at 2026-09-24 01:02:38 on dist-test-slave-5c7l"
server_key: "45baac8825655ede6bf8cc3dab9155c9"
server_key_iv: "78327f136213e71f647bb8edcd736317"
server_key_version: "encryptionkey@0"
I20260924 01:02:38.404026 12923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "103bef62b67e4d409ea1b8a43e868133"
format_stamp: "Formatted at 2026-09-24 01:02:38 on dist-test-slave-5c7l"
server_key: "45baac8825655ede6bf8cc3dab9155c9"
server_key_iv: "78327f136213e71f647bb8edcd736317"
server_key_version: "encryptionkey@0"
I20260924 01:02:38.409475 12923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 01:02:38.412915 12939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:38.414254 12923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:38.414431 12923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "103bef62b67e4d409ea1b8a43e868133"
format_stamp: "Formatted at 2026-09-24 01:02:38 on dist-test-slave-5c7l"
server_key: "45baac8825655ede6bf8cc3dab9155c9"
server_key_iv: "78327f136213e71f647bb8edcd736317"
server_key_version: "encryptionkey@0"
I20260924 01:02:38.414631 12923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:38.439157 12923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:38.443791 12923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:38.444103 12923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:38.445072 12923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:38.446449 12923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:38.446530 12923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:38.446696 12923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:38.446755 12923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:38.477864 12923 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:41525
I20260924 01:02:38.477907 13052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:41525 every 8 connection(s)
I20260924 01:02:38.479207 12923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:38.482000 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 12923
I20260924 01:02:38.482251 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:38.483151 9209 external_mini_cluster.cc:1428] Setting key 6f9086a20f4f74f441d2e61781bb7fe3
Sep 24 01:02:38 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:02:38.503373 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:38.482594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:50373 (local address 127.8.254.126:39625)
0924 01:02:38.483258 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:38.483264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:38.484154 (+ 890us) server_negotiation.cc:407] Connection header received
0924 01:02:38.486783 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:38.486791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:38.486894 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:38.487015 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:38.488476 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.489644 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.491435 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.492218 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.492356 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:38.496051 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:38.496060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:38.496068 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:38.496115 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:38.499801 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:38.500558 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:38.500566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:38.500570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:38.500723 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:38.501150 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:38.501156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:38.501161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:38.501672 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:38.501835 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:38.502673 (+ 838us) server_negotiation.cc:299] Negotiation successful
0924 01:02:38.502929 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":413,"threads_started":1}
I20260924 01:02:38.504127 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:38.483167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39625 (local address 127.8.254.65:50373)
0924 01:02:38.484012 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:38.484058 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:38.486517 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:38.487219 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:38.487227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:38.487746 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:38.488301 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:38.488320 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.489840 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.489844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:38.491280 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:38.491294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.492439 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.492444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:38.492641 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:38.493377 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:38.493399 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:38.495797 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:38.499971 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:38.499977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:38.499991 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:38.500414 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:38.500843 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:38.500846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:38.500848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:38.501022 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:38.501824 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:38.501830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:38.502151 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 01:02:38.502495 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 01:02:38.502881 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":346,"threads_started":1}
I20260924 01:02:38.505812 13053 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:02:38.506145 13053 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:38.507105 13053 heartbeater.cc:511] Master 127.8.254.126:39625 requested a full tablet report, sending...
I20260924 01:02:38.509449 12862 ts_manager.cc:194] Registered new tserver with Master: 103bef62b67e4d409ea1b8a43e868133 (127.8.254.65:41525)
I20260924 01:02:38.510982 12862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:50373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:38.533281 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39625
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:38.850476 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:38.850766 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:38.850967 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:38.858247 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:38.858356 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:38.858537 13059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:38.868177 13059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:38.869669 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:38.870992 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:38.884106 13067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:38.884131 13064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:38.884718 13065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:38.884889 13059 server_base.cc:1034] running on GCE node
I20260924 01:02:38.885527 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:38.886508 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:38.887751 13059 hybrid_clock.cc:648] HybridClock initialized: now 1790211758887703 us; error 57 us; skew 500 ppm
I20260924 01:02:38.888121 13059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:38 dist-test-slave-5c7l krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:38.891878 13059 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:38.893385 13059 webserver.cc:460] Webserver started at http://127.8.254.66:36775/ using document root <none> and password file <none>
I20260924 01:02:38.894075 13059 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:38.894205 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:38.894552 13059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:38.897431 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "2a676cde58c24c2d932594ad5e0ed294"
format_stamp: "Formatted at 2026-09-24 01:02:38 on dist-test-slave-5c7l"
server_key: "08c0f68edac8a45dbd41104bbf5b9e12"
server_key_iv: "3d7763e194979dc638919cbe1e7c333d"
server_key_version: "encryptionkey@0"
I20260924 01:02:38.898257 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "2a676cde58c24c2d932594ad5e0ed294"
format_stamp: "Formatted at 2026-09-24 01:02:38 on dist-test-slave-5c7l"
server_key: "08c0f68edac8a45dbd41104bbf5b9e12"
server_key_iv: "3d7763e194979dc638919cbe1e7c333d"
server_key_version: "encryptionkey@0"
I20260924 01:02:38.903465 13059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 01:02:38.906881 13074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:38.908156 13059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:38.908337 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "2a676cde58c24c2d932594ad5e0ed294"
format_stamp: "Formatted at 2026-09-24 01:02:38 on dist-test-slave-5c7l"
server_key: "08c0f68edac8a45dbd41104bbf5b9e12"
server_key_iv: "3d7763e194979dc638919cbe1e7c333d"
server_key_version: "encryptionkey@0"
I20260924 01:02:38.908545 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:38.927527 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:38.932006 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:38.932365 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:38.933416 13059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:38.934986 13059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:38.935063 13059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:38.935245 13059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:38.935294 13059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:38.969137 13059 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:39117
I20260924 01:02:38.969213 13187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:39117 every 8 connection(s)
I20260924 01:02:38.970537 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:38.971684 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13059
I20260924 01:02:38.971872 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:38.972438 9209 external_mini_cluster.cc:1428] Setting key 22eadca4f0e28e77976b3a619571b438
Sep 24 01:02:38 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:02:38.993234 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:38.973376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:33911 (local address 127.8.254.126:39625)
0924 01:02:38.973608 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:38.973616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:38.975127 (+ 1511us) server_negotiation.cc:407] Connection header received
0924 01:02:38.977101 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:38.977110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:38.977206 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:38.977320 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:38.978920 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.979934 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.982569 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.982977 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.983076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:38.987381 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:38.987387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:38.987390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:38.987426 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:38.990029 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:38.990827 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:38.990837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:38.990839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:38.990966 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:38.991430 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:38.991435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:38.991438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:38.991742 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:38.991912 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:38.992718 (+ 806us) server_negotiation.cc:299] Negotiation successful
0924 01:02:38.992979 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:38.994457 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:38.974024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39625 (local address 127.8.254.66:33911)
0924 01:02:38.974940 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:38.974997 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:38.976765 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:38.977526 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:38.977535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:38.978100 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:38.978744 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:38.978764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.980094 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.980101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:38.982401 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:38.982421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:38.983226 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:38.983246 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:38.983579 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:38.984311 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:38.984351 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:38.987111 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:38.990181 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:38.990191 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:38.990239 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:38.990683 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:38.991085 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:38.991090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:38.991094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:38.991289 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:38.991917 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:38.991926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:38.992352 (+ 426us) client_negotiation.cc:769] Sending connection context
0924 01:02:38.992602 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 01:02:38.993067 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":402,"threads_started":1}
I20260924 01:02:38.996296 13188 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39625
I20260924 01:02:38.996681 13188 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:38.997668 13188 heartbeater.cc:511] Master 127.8.254.126:39625 requested a full tablet report, sending...
I20260924 01:02:38.999666 12862 ts_manager.cc:194] Registered new tserver with Master: 2a676cde58c24c2d932594ad5e0ed294 (127.8.254.66:39117)
I20260924 01:02:39.000924 12862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:33911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:39.021489 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39625
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:39.340402 13193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:39.340754 13193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:39.340930 13193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:39.348325 13193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:39.348481 13193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:39.348673 13193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:39.358808 13193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:39.360203 13193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:39.361536 13193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:39.375869 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:39.375942 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:39.375964 13198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:39.376637 13193 server_base.cc:1034] running on GCE node
I20260924 01:02:39.377349 13193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:39.378485 13193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:39.379736 13193 hybrid_clock.cc:648] HybridClock initialized: now 1790211759379668 us; error 59 us; skew 500 ppm
I20260924 01:02:39.380100 13193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:39.383373 13193 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:39.384888 13193 webserver.cc:460] Webserver started at http://127.8.254.67:35679/ using document root <none> and password file <none>
I20260924 01:02:39.385607 13193 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:39.385756 13193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:39.386154 13193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:39.389269 13193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "bfaa2cc40ff341ce958a340c47544b35"
format_stamp: "Formatted at 2026-09-24 01:02:39 on dist-test-slave-5c7l"
server_key: "541f2876ff1273097b7503a3408a522d"
server_key_iv: "d33eca2432ec0529eb283767a5bd8875"
server_key_version: "encryptionkey@0"
I20260924 01:02:39.390120 13193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "bfaa2cc40ff341ce958a340c47544b35"
format_stamp: "Formatted at 2026-09-24 01:02:39 on dist-test-slave-5c7l"
server_key: "541f2876ff1273097b7503a3408a522d"
server_key_iv: "d33eca2432ec0529eb283767a5bd8875"
server_key_version: "encryptionkey@0"
I20260924 01:02:39.395496 13193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 01:02:39.398928 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:39.400306 13193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:39.400480 13193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "bfaa2cc40ff341ce958a340c47544b35"
format_stamp: "Formatted at 2026-09-24 01:02:39 on dist-test-slave-5c7l"
server_key: "541f2876ff1273097b7503a3408a522d"
server_key_iv: "d33eca2432ec0529eb283767a5bd8875"
server_key_version: "encryptionkey@0"
I20260924 01:02:39.400699 13193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:39.413023 13193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:39.416970 13193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:39.417259 13193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:39.418363 13193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:39.419741 13193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:39.419843 13193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:39.419994 13193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:39.420068 13193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:39.453645 13193 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:40533
I20260924 01:02:39.453681 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:40533 every 8 connection(s)
I20260924 01:02:39.455065 13193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:39.461376 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13193
I20260924 01:02:39.461586 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:39.462035 9209 external_mini_cluster.cc:1428] Setting key 7e35025cd5385923515f29896aa07807
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:39.474941 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.457801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:59733 (local address 127.8.254.126:39625)
0924 01:02:39.457978 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:39.457983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:39.459178 (+ 1195us) server_negotiation.cc:407] Connection header received
0924 01:02:39.460957 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:39.460962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:39.461091 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:39.461224 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:39.462870 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.463636 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.465141 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.465536 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.465674 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.469048 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:39.469053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:39.469056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:39.469090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:39.471820 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.472654 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.472658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.472660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.472785 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.473228 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.473244 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.473247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.473506 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:39.473664 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:39.474378 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 01:02:39.474592 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:02:39.476023 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.458322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39625 (local address 127.8.254.67:59733)
0924 01:02:39.459006 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:39.459042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:39.460654 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:39.461420 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:39.461429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:39.462047 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:39.462682 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:39.462699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.463800 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.463805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:39.464993 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:39.465008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.465770 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.465777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:39.466000 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.466719 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:39.466747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:39.468831 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:39.471966 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:39.471976 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:39.471993 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:39.472518 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:39.472899 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:39.472904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:39.472908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:39.473095 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:39.473637 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:39.473644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:39.473981 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 01:02:39.474270 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 01:02:39.474806 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":265,"threads_started":1}
I20260924 01:02:39.477638 13322 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39625
I20260924 01:02:39.477946 13322 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:39.478842 13322 heartbeater.cc:511] Master 127.8.254.126:39625 requested a full tablet report, sending...
I20260924 01:02:39.480309 12861 ts_manager.cc:194] Registered new tserver with Master: bfaa2cc40ff341ce958a340c47544b35 (127.8.254.67:40533)
I20260924 01:02:39.481158 12861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:59733
I20260924 01:02:39.495792 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:02:39.514398 13053 heartbeater.cc:503] Master 127.8.254.126:39625 was elected leader, sending a full tablet report...
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:39.544387 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.528896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.8.254.65:41525)
0924 01:02:39.529503 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:39.529511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:39.529538 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:02:39.529603 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:39.529609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:39.529823 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:39.529994 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:39.531434 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.532536 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.533884 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.534547 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.534650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.538644 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:39.538656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:39.538672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:39.538713 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:39.541468 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.542146 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.542153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.542155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.542348 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.542732 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.542739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.542742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.543102 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:39.543409 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:39.543731 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 01:02:39.544021 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":319,"threads_started":1}
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:39.562718 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.548922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.8.254.126:39625)
0924 01:02:39.549102 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:39.549106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:39.549320 (+ 214us) server_negotiation.cc:407] Connection header received
0924 01:02:39.549467 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:39.549471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:39.549548 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:39.549631 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:39.551015 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.551744 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.552933 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.553350 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.553455 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.557862 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:39.557868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:39.557871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:39.557905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:39.560324 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.561027 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.561030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.561032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.561160 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.561474 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.561477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.561479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.561795 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:39.561979 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:39.562309 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 01:02:39.562478 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 01:02:39.563338 12861 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38900
I20260924 01:02:39.564409 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12923
I20260924 01:02:39.577347 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.565403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.8.254.65:41525)
0924 01:02:39.565616 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:39.565622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:39.565748 (+ 126us) server_negotiation.cc:407] Connection header received
0924 01:02:39.565954 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:39.565964 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:39.566057 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:39.566164 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:39.567392 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.568086 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.569324 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.569753 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.569890 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.572305 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:39.572310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:39.572314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:39.572348 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:39.574798 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.575479 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.575488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.575491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.575610 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.575984 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.575989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.575992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.576260 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:39.576506 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:39.576795 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 01:02:39.577028 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 01:02:39.577978 13048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:39.661875 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12923
I20260924 01:02:39.679163 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13059
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:39.695597 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.680158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.8.254.66:39117)
0924 01:02:39.680692 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:39.680699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:39.680724 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:02:39.680791 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:39.680797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:39.681018 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:39.681196 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:39.682703 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.683867 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.685259 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.685811 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.685971 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.690258 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:39.690282 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:39.690306 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:39.690358 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:39.693137 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.693771 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.693778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.693780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.693906 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.694274 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.694278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.694281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.694632 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:39.694884 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:39.695118 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 01:02:39.695310 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":268,"threads_started":1}
I20260924 01:02:39.696856 13183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:39.780184 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13059
I20260924 01:02:39.797197 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13193
Sep 24 01:02:39 dist-test-slave-5c7l krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:39.822937 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:39.798326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.8.254.67:40533)
0924 01:02:39.798965 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:39.798972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:39.799000 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:02:39.799059 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:39.799065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:39.799303 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:39.799430 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:39.800675 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.801776 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.803170 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:39.803912 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:39.804030 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:39.817338 (+ 13308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:39.817353 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:39.817370 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:39.817411 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:39.820166 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.820875 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.820883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.820886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.821027 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:39.821346 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:39.821349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:39.821351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:39.821694 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:39.821986 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:39.822315 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 01:02:39.822522 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":320,"threads_started":1}
I20260924 01:02:39.824254 13317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:39.907449 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13193
I20260924 01:02:39.924023 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 12830
I20260924 01:02:39.925201 12892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:40.030226 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 12830
2026-09-24T01:02:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3846 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:40 dist-test-slave-5c7l krb5kdc[13341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:40 dist-test-slave-5c7l krb5kdc[13341](info): set up 2 sockets
Sep 24 01:02:40 dist-test-slave-5c7l krb5kdc[13341](info): commencing operation
krb5kdc: starting...
W20260924 01:02:41.115679 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:41 dist-test-slave-5c7l krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:41.318521 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:34027
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:41.644758 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:41.645186 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:41.645382 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:41.652864 13354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:41.653036 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:41.653151 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:41.653275 13354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:41.653369 13354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:41.663621 13354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:41.665040 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:41.666448 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:41.678507 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:41.678542 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:41.678519 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:41.679198 13354 server_base.cc:1034] running on GCE node
I20260924 01:02:41.680120 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:41.681591 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:41.682813 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790211761682763 us; error 48 us; skew 500 ppm
I20260924 01:02:41.683146 13354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:41 dist-test-slave-5c7l krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:41.686339 13354 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:41.687626 13354 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:41.687742 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:41.688097 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:41.691197 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "f66544e187884521ac2678cba8df88a2"
format_stamp: "Formatted at 2026-09-24 01:02:41 on dist-test-slave-5c7l"
server_key: "1df35cd573735cc77e4b1b426c039331"
server_key_iv: "ece42f74f40438ea1bbcc8f15b1611e2"
server_key_version: "encryptionkey@0"
I20260924 01:02:41.692013 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "f66544e187884521ac2678cba8df88a2"
format_stamp: "Formatted at 2026-09-24 01:02:41 on dist-test-slave-5c7l"
server_key: "1df35cd573735cc77e4b1b426c039331"
server_key_iv: "ece42f74f40438ea1bbcc8f15b1611e2"
server_key_version: "encryptionkey@0"
I20260924 01:02:41.697134 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:41.700664 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:41.701928 13354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:41.702109 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "f66544e187884521ac2678cba8df88a2"
format_stamp: "Formatted at 2026-09-24 01:02:41 on dist-test-slave-5c7l"
server_key: "1df35cd573735cc77e4b1b426c039331"
server_key_iv: "ece42f74f40438ea1bbcc8f15b1611e2"
server_key_version: "encryptionkey@0"
I20260924 01:02:41.702405 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:41.723397 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:41.727486 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:41.727772 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:41.747164 13354 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:34027
I20260924 01:02:41.747139 13419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:34027 every 8 connection(s)
I20260924 01:02:41.748117 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:41.751788 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13354
I20260924 01:02:41.752012 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:41.752420 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:41.752861 9209 external_mini_cluster.cc:1428] Setting key 37d976ff595976ed546131684629b91b
Sep 24 01:02:41 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:41.768582 13420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2: Bootstrap starting.
I20260924 01:02:41.772280 13420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:41.773572 13420 log.cc:824] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:41.777089 13420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2: No bootstrap required, opened a new log
I20260924 01:02:41.777597 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:41.755118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.8.254.126:34027)
0924 01:02:41.755917 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:41.755930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:41.755970 (+ 40us) server_negotiation.cc:407] Connection header received
0924 01:02:41.758189 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:41.758283 (+ 94us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:41.758693 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:41.759035 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:41.760946 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:41.762294 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:41.764082 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:41.764801 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:41.764938 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:41.769821 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:41.769831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:41.769845 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:41.769891 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:41.773062 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:41.773899 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:41.773906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:41.773913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:41.774074 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:41.774580 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:41.774588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:41.774593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:41.775321 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:41.775548 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:41.776094 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 01:02:41.776492 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":376,"threads_started":1}
I20260924 01:02:41.784627 13420 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66544e187884521ac2678cba8df88a2" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 34027 } }
I20260924 01:02:41.785066 13420 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:41.785243 13420 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66544e187884521ac2678cba8df88a2, State: Initialized, Role: FOLLOWER
I20260924 01:02:41.786324 13420 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66544e187884521ac2678cba8df88a2" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 34027 } }
I20260924 01:02:41.786660 13420 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:41.786839 13420 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:41.787053 13420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:41.791221 13420 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66544e187884521ac2678cba8df88a2" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 34027 } }
I20260924 01:02:41.791962 13420 leader_election.cc:302] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66544e187884521ac2678cba8df88a2; no voters:
I20260924 01:02:41.792574 13420 leader_election.cc:288] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:41.792773 13425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:41.793761 13425 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [term 1 LEADER]: Becoming Leader. State: Replica: f66544e187884521ac2678cba8df88a2, State: Running, Role: LEADER
I20260924 01:02:41.794368 13425 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66544e187884521ac2678cba8df88a2" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 34027 } }
I20260924 01:02:41.795651 13420 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:41.799935 13426 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66544e187884521ac2678cba8df88a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66544e187884521ac2678cba8df88a2" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 34027 } } }
I20260924 01:02:41.800457 13426 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:41.800977 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66544e187884521ac2678cba8df88a2. Latest consensus state: current_term: 1 leader_uuid: "f66544e187884521ac2678cba8df88a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66544e187884521ac2678cba8df88a2" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 34027 } } }
I20260924 01:02:41.801299 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:41.801481 13433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:41.811057 13433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:41.812466 13442 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 01:02:41.821565 13433 catalog_manager.cc:1347] Generated new cluster ID: 88f658e7b6374a229adc30a8bfaf8d16
I20260924 01:02:41.821698 13433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:41.840693 13433 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:41.842440 13433 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:41.854169 13433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2: Generated new TSK 0
I20260924 01:02:41.855150 13433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:41.910701 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:34027
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:42.244660 13446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:42.244963 13446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:42.245062 13446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:42.252421 13446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:42.252547 13446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:42.252667 13446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:42.262460 13446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:42.263763 13446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:42.264999 13446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:42.278082 13451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:42.278126 13454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:42.278103 13452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:42.279405 13446 server_base.cc:1034] running on GCE node
I20260924 01:02:42.280052 13446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:42.281038 13446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:42.282301 13446 hybrid_clock.cc:648] HybridClock initialized: now 1790211762282240 us; error 43 us; skew 500 ppm
I20260924 01:02:42.282657 13446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:42 dist-test-slave-5c7l krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:42.286019 13446 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:42.287268 13446 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:42.287412 13446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:42.287740 13446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:42.290758 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "27cfd0de42944dffaeb3ef5548bff3a5"
format_stamp: "Formatted at 2026-09-24 01:02:42 on dist-test-slave-5c7l"
server_key: "1fef5dcdb8efe48d9c84d634e20c4bfa"
server_key_iv: "70e6c189a585059ff6caf32ba1847b4b"
server_key_version: "encryptionkey@0"
I20260924 01:02:42.291590 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "27cfd0de42944dffaeb3ef5548bff3a5"
format_stamp: "Formatted at 2026-09-24 01:02:42 on dist-test-slave-5c7l"
server_key: "1fef5dcdb8efe48d9c84d634e20c4bfa"
server_key_iv: "70e6c189a585059ff6caf32ba1847b4b"
server_key_version: "encryptionkey@0"
I20260924 01:02:42.296940 13446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:42.300493 13460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:42.301865 13446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 01:02:42.302055 13446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "27cfd0de42944dffaeb3ef5548bff3a5"
format_stamp: "Formatted at 2026-09-24 01:02:42 on dist-test-slave-5c7l"
server_key: "1fef5dcdb8efe48d9c84d634e20c4bfa"
server_key_iv: "70e6c189a585059ff6caf32ba1847b4b"
server_key_version: "encryptionkey@0"
I20260924 01:02:42.302376 13446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:42.312980 13446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:42.317200 13446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:42.317481 13446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:42.318527 13446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:42.319830 13446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:42.319953 13446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:42.320088 13446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:42.320188 13446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:42.354827 13446 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:41357
I20260924 01:02:42.354923 13572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:41357 every 8 connection(s)
I20260924 01:02:42.355796 13446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:42.362870 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13446
I20260924 01:02:42.363090 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:42.363898 9209 external_mini_cluster.cc:1428] Setting key 35c577e792c5cea7b6aefc1ec82661d0
Sep 24 01:02:42 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:42.380070 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:42.358425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:53007 (local address 127.8.254.126:34027)
0924 01:02:42.358875 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:42.358880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:42.359963 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 01:02:42.361815 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:42.361821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:42.361917 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:42.362034 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:42.364240 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.365126 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.367499 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.367955 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.368066 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:42.372653 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:42.372662 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:42.372668 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:42.372715 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:42.376425 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:42.377289 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:42.377297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:42.377304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:42.377466 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:42.377901 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:42.377909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:42.377915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:42.378326 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:42.378474 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:42.379298 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 01:02:42.379552 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":236,"threads_started":1}
I20260924 01:02:42.381094 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:42.359071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:34027 (local address 127.8.254.65:53007)
0924 01:02:42.359780 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:42.359827 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:42.361520 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:42.362339 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:42.362369 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:42.363152 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:42.364021 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:42.364049 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.365315 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.365318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:42.367313 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:42.367333 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.368208 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.368217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:42.368563 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:42.369352 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:42.369391 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:42.372363 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:42.376615 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:42.376623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:42.376638 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:42.377139 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:42.377581 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:42.377585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:42.377588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:42.377776 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:42.378475 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:42.378483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:42.378867 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 01:02:42.379144 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 01:02:42.379617 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":287,"threads_started":1}
I20260924 01:02:42.382951 13573 heartbeater.cc:347] Connected to a master server at 127.8.254.126:34027
I20260924 01:02:42.383309 13573 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:42.383992 13573 heartbeater.cc:511] Master 127.8.254.126:34027 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:02:42.386251 13385 ts_manager.cc:194] Registered new tserver with Master: 27cfd0de42944dffaeb3ef5548bff3a5 (127.8.254.65:41357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:02:42.387940 13385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:53007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:42.411849 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:34027
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:42.733675 13579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:42.734004 13579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:42.734182 13579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:42.741585 13579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:42.741751 13579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:42.741951 13579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:42.752122 13579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:42.753485 13579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:42.754936 13579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:42.767764 13584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:42.767817 13587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:42.767859 13585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:42.768818 13579 server_base.cc:1034] running on GCE node
I20260924 01:02:42.769374 13579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:42.770380 13579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:42.771589 13579 hybrid_clock.cc:648] HybridClock initialized: now 1790211762771512 us; error 65 us; skew 500 ppm
I20260924 01:02:42.771883 13579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:42 dist-test-slave-5c7l krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:42.774977 13579 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:42.776103 13579 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:42.776199 13579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:42.776465 13579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:42.779517 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "bd3f6823b0644a36811ceb835e718ac3"
format_stamp: "Formatted at 2026-09-24 01:02:42 on dist-test-slave-5c7l"
server_key: "25493521c4283e9633ac46183ab6893f"
server_key_iv: "40a3ab87952bbeea8a702761fe6cb2e1"
server_key_version: "encryptionkey@0"
I20260924 01:02:42.780308 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "bd3f6823b0644a36811ceb835e718ac3"
format_stamp: "Formatted at 2026-09-24 01:02:42 on dist-test-slave-5c7l"
server_key: "25493521c4283e9633ac46183ab6893f"
server_key_iv: "40a3ab87952bbeea8a702761fe6cb2e1"
server_key_version: "encryptionkey@0"
I20260924 01:02:42.785887 13579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:42.789783 13593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:42.791361 13579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 01:02:42.791545 13579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "bd3f6823b0644a36811ceb835e718ac3"
format_stamp: "Formatted at 2026-09-24 01:02:42 on dist-test-slave-5c7l"
server_key: "25493521c4283e9633ac46183ab6893f"
server_key_iv: "40a3ab87952bbeea8a702761fe6cb2e1"
server_key_version: "encryptionkey@0"
I20260924 01:02:42.791772 13579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:42.805399 13579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:42.810026 13579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:42.810343 13579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:42.811316 13579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:42.812649 13579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:42.812774 13579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:42.812911 13579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:42.813006 13579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:42.846494 13579 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:36787
I20260924 01:02:42.846567 13705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:36787 every 8 connection(s)
I20260924 01:02:42.847458 13579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:42.850164 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13579
I20260924 01:02:42.850364 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:42.850981 9209 external_mini_cluster.cc:1428] Setting key 0f631f0bee0214bc19866c32109ca315
Sep 24 01:02:42 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:42.869503 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:42.849908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:36635 (local address 127.8.254.126:34027)
0924 01:02:42.850080 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:42.850084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:42.851437 (+ 1353us) server_negotiation.cc:407] Connection header received
0924 01:02:42.853487 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:42.853492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:42.853576 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:42.853697 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:42.855372 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.856084 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.857663 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.858081 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.858183 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:42.862445 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:42.862454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:42.862462 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:42.862509 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:42.866070 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:42.866946 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:42.866954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:42.866961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:42.867118 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:42.867557 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:42.867565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:42.867571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:42.867998 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:42.868172 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:42.868840 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 01:02:42.869094 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:02:42.870281 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:42.850526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:34027 (local address 127.8.254.66:36635)
0924 01:02:42.851274 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:42.851318 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:42.853226 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:42.853935 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:42.853943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:42.854632 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:42.855199 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:42.855215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.856256 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.856261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:42.857491 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:42.857502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:42.858358 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:42.858365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:42.858689 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:42.859390 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:42.859424 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:42.862154 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:42.866263 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:42.866271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:42.866284 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:42.866802 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:42.867242 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:42.867246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:42.867248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:42.867429 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:42.868122 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:42.868128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:42.868454 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 01:02:42.868678 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 01:02:42.869070 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:02:42.871868 13706 heartbeater.cc:347] Connected to a master server at 127.8.254.126:34027
I20260924 01:02:42.872179 13706 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:42.873014 13706 heartbeater.cc:511] Master 127.8.254.126:34027 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:42.874713 13385 ts_manager.cc:194] Registered new tserver with Master: bd3f6823b0644a36811ceb835e718ac3 (127.8.254.66:36787)
I20260924 01:02:42.875524 13385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:36635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:42.898012 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:34027
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:43.218271 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:43.218688 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:43.218878 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:43.226052 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:43.226156 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:43.226399 13711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:43.236181 13711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:43.237555 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:43.238993 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:43.252178 13716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:43.252125 13717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:43.252149 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:43.253114 13711 server_base.cc:1034] running on GCE node
I20260924 01:02:43.253858 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:43.254931 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:43.256142 13711 hybrid_clock.cc:648] HybridClock initialized: now 1790211763256102 us; error 29 us; skew 500 ppm
I20260924 01:02:43.256492 13711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:43 dist-test-slave-5c7l krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:43.259616 13711 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:43.260782 13711 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:43.260922 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:43.261236 13711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:43.264235 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "c2a6d1a2de7844459b195943560fadf2"
format_stamp: "Formatted at 2026-09-24 01:02:43 on dist-test-slave-5c7l"
server_key: "e1ef408416565d24d1385f19bc7bd02c"
server_key_iv: "548a6efc68254cdb4fa138290ef61b21"
server_key_version: "encryptionkey@0"
I20260924 01:02:43.265074 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "c2a6d1a2de7844459b195943560fadf2"
format_stamp: "Formatted at 2026-09-24 01:02:43 on dist-test-slave-5c7l"
server_key: "e1ef408416565d24d1385f19bc7bd02c"
server_key_iv: "548a6efc68254cdb4fa138290ef61b21"
server_key_version: "encryptionkey@0"
I20260924 01:02:43.270337 13711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:02:43.273926 13725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:43.275447 13711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:43.275650 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "c2a6d1a2de7844459b195943560fadf2"
format_stamp: "Formatted at 2026-09-24 01:02:43 on dist-test-slave-5c7l"
server_key: "e1ef408416565d24d1385f19bc7bd02c"
server_key_iv: "548a6efc68254cdb4fa138290ef61b21"
server_key_version: "encryptionkey@0"
I20260924 01:02:43.275905 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:43.289937 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:43.294000 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:43.294339 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:43.295346 13711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:43.296876 13711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:43.296996 13711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:43.297113 13711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:43.297200 13711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:43.330127 13711 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:44283
I20260924 01:02:43.330180 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:44283 every 8 connection(s)
I20260924 01:02:43.331079 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:43.337072 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13711
I20260924 01:02:43.337297 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:43.337857 9209 external_mini_cluster.cc:1428] Setting key cbc56aae3c7c770efb1275339651fa06
Sep 24 01:02:43 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:43.351778 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.333674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:35215 (local address 127.8.254.126:34027)
0924 01:02:43.333863 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.333867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.335015 (+ 1148us) server_negotiation.cc:407] Connection header received
0924 01:02:43.336921 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.336925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.337031 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.337121 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:43.339276 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.340273 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.342131 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.342614 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.342715 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.346350 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:43.346355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:43.346358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:43.346395 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:43.348839 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.349609 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.349613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.349615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.349737 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.350115 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.350119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.350121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.350443 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:43.350553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.351264 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.351478 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 01:02:43.352903 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.334135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:34027 (local address 127.8.254.67:35215)
0924 01:02:43.334851 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.334898 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.336553 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.337430 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.337442 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.338238 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:43.339027 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.339047 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.340445 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.340449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.341989 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.342002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.342854 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.342860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.343055 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.343788 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:43.343815 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:43.346100 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:43.349024 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:43.349032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:43.349046 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:43.349487 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:43.349835 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:43.349839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:43.349841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:43.350013 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:43.350556 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:43.350562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:43.350903 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.351134 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.351542 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":277,"threads_started":1}
I20260924 01:02:43.354526 13838 heartbeater.cc:347] Connected to a master server at 127.8.254.126:34027
I20260924 01:02:43.354843 13838 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:43.355695 13838 heartbeater.cc:511] Master 127.8.254.126:34027 requested a full tablet report, sending...
I20260924 01:02:43.357236 13385 ts_manager.cc:194] Registered new tserver with Master: c2a6d1a2de7844459b195943560fadf2 (127.8.254.67:44283)
I20260924 01:02:43.358232 13385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:35215
I20260924 01:02:43.365926 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:02:43.382330 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.370359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.8.254.126:34027)
0924 01:02:43.370524 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.370528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.371084 (+ 556us) server_negotiation.cc:407] Connection header received
0924 01:02:43.371213 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.371217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.371283 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.371365 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:43.372719 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.373509 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.374764 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.375175 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.375271 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.377438 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:43.377443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:43.377446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:43.377478 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:43.380029 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.380654 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.380658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.380661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.380781 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.381103 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.381106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.381108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.381431 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:43.381610 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.381890 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.382047 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:02:43.390981 13573 heartbeater.cc:503] Master 127.8.254.126:34027 was elected leader, sending a full tablet report...
I20260924 01:02:43.386356 13385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:43.393461 13385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:43.431265 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.406019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:41357 (local address 127.0.0.1:54482)
0924 01:02:43.418420 (+ 12401us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.418492 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.418749 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.419886 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.419897 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.419933 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.420413 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.420431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.423235 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.423244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.427163 (+ 3919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.427188 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.429826 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.429836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.430146 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.430246 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.430414 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.430685 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12029,"thread_start_us":11886,"threads_started":1}
I20260924 01:02:43.433872 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.406267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.8.254.65:41357)
0924 01:02:43.417909 (+ 11642us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.417921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.419131 (+ 1210us) server_negotiation.cc:407] Connection header received
0924 01:02:43.419269 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.419280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.419635 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.420631 (+ 996us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.420862 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.423034 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.427468 (+ 4434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.429541 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.432673 (+ 3132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.433029 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.433122 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.433399 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11410,"thread_start_us":11247,"threads_started":1}
I20260924 01:02:43.435359 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.406297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:44283 (local address 127.0.0.1:59864)
0924 01:02:43.419872 (+ 13575us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.419899 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.420023 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.421586 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.421594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.421629 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.423800 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.423821 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.427916 (+ 4095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.427924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.432151 (+ 4227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.432172 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.434487 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.434493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.434689 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.434720 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.434819 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.434962 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13417,"spinlock_wait_cycles":2944,"thread_start_us":13822,"threads_started":1}
I20260924 01:02:43.437326 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.406612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.8.254.67:44283)
0924 01:02:43.420590 (+ 13978us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.420600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.420636 (+ 36us) server_negotiation.cc:407] Connection header received
0924 01:02:43.420740 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.420747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.421007 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.421239 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.424086 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.427666 (+ 3580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.432376 (+ 4710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.433935 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.434042 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.434297 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.436833 (+ 2536us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.437020 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13812,"thread_start_us":13655,"threads_started":1}
I20260924 01:02:43.438957 13508 tablet_service.cc:1467] Processing CreateTablet for tablet 194512549c9d4496ae508a6cc0cafe21 (DEFAULT_TABLE table=test-table [id=1520038f8cfa452787648bbb3fc15e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:43.440516 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.420721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:36787 (local address 127.0.0.1:46142)
0924 01:02:43.432275 (+ 11554us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.432299 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.432452 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.433850 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.433856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.433886 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.434069 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.434079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.436174 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.436179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.438125 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.438141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.439744 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.439751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.439929 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.439967 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.440095 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.440235 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11378,"thread_start_us":17299,"threads_started":1}
I20260924 01:02:43.440632 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.420823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.8.254.66:36787)
0924 01:02:43.432954 (+ 12131us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.432964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.432994 (+ 30us) server_negotiation.cc:407] Connection header received
0924 01:02:43.433102 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.433111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.433416 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.434276 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.434510 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.435988 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.438372 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.439558 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.439725 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.439912 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.440136 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.440310 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11956,"thread_start_us":12746,"threads_started":1}
I20260924 01:02:43.440843 13508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194512549c9d4496ae508a6cc0cafe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:43.442157 13773 tablet_service.cc:1467] Processing CreateTablet for tablet 194512549c9d4496ae508a6cc0cafe21 (DEFAULT_TABLE table=test-table [id=1520038f8cfa452787648bbb3fc15e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:43.443715 13773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194512549c9d4496ae508a6cc0cafe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:43.444024 13641 tablet_service.cc:1467] Processing CreateTablet for tablet 194512549c9d4496ae508a6cc0cafe21 (DEFAULT_TABLE table=test-table [id=1520038f8cfa452787648bbb3fc15e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:43.445129 13641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194512549c9d4496ae508a6cc0cafe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:43.455986 13856 tablet_bootstrap.cc:492] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Bootstrap starting.
I20260924 01:02:43.459693 13856 tablet_bootstrap.cc:654] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:43.460722 13856 log.cc:824] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:43.462258 13857 tablet_bootstrap.cc:492] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Bootstrap starting.
I20260924 01:02:43.464951 13858 tablet_bootstrap.cc:492] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Bootstrap starting.
I20260924 01:02:43.466039 13856 tablet_bootstrap.cc:492] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: No bootstrap required, opened a new log
I20260924 01:02:43.466369 13856 ts_tablet_manager.cc:1397] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260924 01:02:43.466555 13857 tablet_bootstrap.cc:654] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:43.467677 13857 log.cc:824] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:43.468557 13858 tablet_bootstrap.cc:654] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:43.469763 13858 log.cc:824] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:43.471396 13857 tablet_bootstrap.cc:492] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: No bootstrap required, opened a new log
I20260924 01:02:43.471668 13857 ts_tablet_manager.cc:1397] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 01:02:43.472862 13858 tablet_bootstrap.cc:492] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: No bootstrap required, opened a new log
I20260924 01:02:43.473111 13858 ts_tablet_manager.cc:1397] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 01:02:43.476786 13856 raft_consensus.cc:348] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.477257 13856 raft_consensus.cc:374] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:43.477406 13856 raft_consensus.cc:729] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f6823b0644a36811ceb835e718ac3, State: Initialized, Role: FOLLOWER
I20260924 01:02:43.478075 13856 consensus_queue.cc:260] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.479293 13706 heartbeater.cc:503] Master 127.8.254.126:34027 was elected leader, sending a full tablet report...
I20260924 01:02:43.480135 13857 raft_consensus.cc:348] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.480490 13857 raft_consensus.cc:374] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:43.480635 13857 raft_consensus.cc:729] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cfd0de42944dffaeb3ef5548bff3a5, State: Initialized, Role: FOLLOWER
I20260924 01:02:43.481344 13856 ts_tablet_manager.cc:1428] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 01:02:43.481838 13857 consensus_queue.cc:260] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.483470 13858 raft_consensus.cc:348] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.483923 13858 raft_consensus.cc:374] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:43.484026 13858 raft_consensus.cc:729] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a6d1a2de7844459b195943560fadf2, State: Initialized, Role: FOLLOWER
I20260924 01:02:43.484697 13858 consensus_queue.cc:260] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.485621 13857 ts_tablet_manager.cc:1428] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 01:02:43.487097 13838 heartbeater.cc:503] Master 127.8.254.126:34027 was elected leader, sending a full tablet report...
I20260924 01:02:43.487771 13858 ts_tablet_manager.cc:1428] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260924 01:02:43.537223 13863 raft_consensus.cc:482] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:43.537525 13863 raft_consensus.cc:504] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.539274 13863 leader_election.cc:288] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a6d1a2de7844459b195943560fadf2 (127.8.254.67:44283), bd3f6823b0644a36811ceb835e718ac3 (127.8.254.66:36787)
I20260924 01:02:43.546666 13862 raft_consensus.cc:482] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:43.547027 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.539855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:55597 (local address 127.8.254.67:44283)
0924 01:02:43.540045 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.540050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.540648 (+ 598us) server_negotiation.cc:407] Connection header received
0924 01:02:43.540941 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.540947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.541028 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.541140 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.542012 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.543068 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.545007 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.545871 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.546004 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.546057 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.546594 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.546744 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:02:43.547132 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.539757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:44283 (local address 127.8.254.65:55597)
0924 01:02:43.540526 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.540548 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.540699 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.541292 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.541297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.541323 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.541816 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.541828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.543212 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.543218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.544831 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.544843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.546107 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.546113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.546389 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.546426 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.546558 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.546707 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":4,"spinlock_wait_cycles":26368,"thread_start_us":441,"threads_started":1}
I20260924 01:02:43.547215 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.541029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:54521 (local address 127.8.254.66:36787)
0924 01:02:43.541249 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.541256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.541470 (+ 214us) server_negotiation.cc:407] Connection header received
0924 01:02:43.541647 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.541650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.541726 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.541845 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.542518 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.543445 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.545232 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.546099 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.546279 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.546357 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.546795 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.546938 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 01:02:43.547188 13862 raft_consensus.cc:504] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.547884 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.540701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:36787 (local address 127.8.254.65:54521)
0924 01:02:43.541353 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.541387 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.541531 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.541872 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.541876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.541901 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.542352 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.542364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.543590 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.543594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.545032 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.545048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.546435 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.546441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.546612 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.546644 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.547370 (+ 726us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.547545 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":347,"threads_started":1}
I20260924 01:02:43.548583 13793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194512549c9d4496ae508a6cc0cafe21" candidate_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a6d1a2de7844459b195943560fadf2" is_pre_election: true
I20260924 01:02:43.549002 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194512549c9d4496ae508a6cc0cafe21" candidate_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f6823b0644a36811ceb835e718ac3" is_pre_election: true
I20260924 01:02:43.549176 13793 raft_consensus.cc:2454] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27cfd0de42944dffaeb3ef5548bff3a5 in term 0.
I20260924 01:02:43.549372 13862 leader_election.cc:288] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27cfd0de42944dffaeb3ef5548bff3a5 (127.8.254.65:41357), c2a6d1a2de7844459b195943560fadf2 (127.8.254.67:44283)
I20260924 01:02:43.549497 13661 raft_consensus.cc:2454] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27cfd0de42944dffaeb3ef5548bff3a5 in term 0.
I20260924 01:02:43.550693 13462 leader_election.cc:302] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cfd0de42944dffaeb3ef5548bff3a5, bd3f6823b0644a36811ceb835e718ac3; no voters:
I20260924 01:02:43.551520 13863 raft_consensus.cc:2781] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:43.551774 13863 raft_consensus.cc:482] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:43.551921 13863 raft_consensus.cc:3037] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:43.555212 13863 raft_consensus.cc:504] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.556034 13863 leader_election.cc:288] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [CANDIDATE]: Term 1 election: Requested vote from peers c2a6d1a2de7844459b195943560fadf2 (127.8.254.67:44283), bd3f6823b0644a36811ceb835e718ac3 (127.8.254.66:36787)
I20260924 01:02:43.556607 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194512549c9d4496ae508a6cc0cafe21" candidate_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f6823b0644a36811ceb835e718ac3"
I20260924 01:02:43.556627 13793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194512549c9d4496ae508a6cc0cafe21" candidate_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a6d1a2de7844459b195943560fadf2"
I20260924 01:02:43.556828 13661 raft_consensus.cc:3037] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:43.556833 13793 raft_consensus.cc:3037] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:43.558063 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.549789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:33809 (local address 127.8.254.65:41357)
0924 01:02:43.550031 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.550039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.551155 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 01:02:43.551426 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.551433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.551517 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.551615 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.552166 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.553171 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.555684 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.556999 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.557145 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.557208 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.557737 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.557861 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:02:43.558058 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.550284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:46849 (local address 127.8.254.67:44283)
0924 01:02:43.550488 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.550496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.551510 (+ 1014us) server_negotiation.cc:407] Connection header received
0924 01:02:43.551748 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.551755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.551834 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.551905 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:02:43.552727 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.553658 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.555456 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.556704 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.556885 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.556965 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.557584 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.557748 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 01:02:43.558820 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.549505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:41357 (local address 127.8.254.66:33809)
0924 01:02:43.551013 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.551055 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.551278 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.551652 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.551658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.551685 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.552016 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.552030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.553311 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.553316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.555515 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.555531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.557372 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.557378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.557576 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.557606 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.558243 (+ 637us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.558407 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"thread_start_us":1598,"threads_started":1}
I20260924 01:02:43.558907 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.550044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:44283 (local address 127.8.254.66:46849)
0924 01:02:43.551424 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:43.551443 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:43.551602 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:43.552103 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:43.552108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:43.552134 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:02:43.552553 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.552568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.553788 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.553795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.555294 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:43.555306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.557131 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.557138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:43.557369 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.557417 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 01:02:43.558370 (+ 953us) client_negotiation.cc:240] Negotiation successful
0924 01:02:43.558486 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"mutex_wait_us":3,"thread_start_us":610,"threads_started":1}
I20260924 01:02:43.559783 13528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194512549c9d4496ae508a6cc0cafe21" candidate_uuid: "bd3f6823b0644a36811ceb835e718ac3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" is_pre_election: true
I20260924 01:02:43.559818 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194512549c9d4496ae508a6cc0cafe21" candidate_uuid: "bd3f6823b0644a36811ceb835e718ac3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a6d1a2de7844459b195943560fadf2" is_pre_election: true
I20260924 01:02:43.560268 13528 raft_consensus.cc:2379] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd3f6823b0644a36811ceb835e718ac3 in current term 1: Already voted for candidate 27cfd0de42944dffaeb3ef5548bff3a5 in this term.
I20260924 01:02:43.560246 13793 raft_consensus.cc:2454] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27cfd0de42944dffaeb3ef5548bff3a5 in term 1.
I20260924 01:02:43.560648 13462 leader_election.cc:302] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cfd0de42944dffaeb3ef5548bff3a5, c2a6d1a2de7844459b195943560fadf2; no voters:
I20260924 01:02:43.560969 13863 raft_consensus.cc:2781] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:43.561117 13594 leader_election.cc:302] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd3f6823b0644a36811ceb835e718ac3; no voters: 27cfd0de42944dffaeb3ef5548bff3a5, c2a6d1a2de7844459b195943560fadf2
I20260924 01:02:43.561493 13863 raft_consensus.cc:686] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 1 LEADER]: Becoming Leader. State: Replica: 27cfd0de42944dffaeb3ef5548bff3a5, State: Running, Role: LEADER
I20260924 01:02:43.561955 13661 raft_consensus.cc:2454] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27cfd0de42944dffaeb3ef5548bff3a5 in term 1.
I20260924 01:02:43.562168 13862 raft_consensus.cc:2726] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 01:02:43.562466 13863 consensus_queue.cc:237] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } }
I20260924 01:02:43.567056 13385 catalog_manager.cc:5495] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27cfd0de42944dffaeb3ef5548bff3a5 (127.8.254.65). New cstate: current_term: 1 leader_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cfd0de42944dffaeb3ef5548bff3a5" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 } health_report { overall_health: UNKNOWN } } }
W20260924 01:02:43.582583 13839 tablet.cc:2369] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:02:43.585080 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.579599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.8.254.65:41357)
0924 01:02:43.579810 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.579816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.579835 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:02:43.580020 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.580024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.580131 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.580300 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:02:43.580670 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.581424 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.582901 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.583318 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.583452 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.583942 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:02:43.584452 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:02:43.584615 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.584696 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.584791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:02:43.593039 13793 raft_consensus.cc:1261] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 1 FOLLOWER]: Refusing update from remote peer 27cfd0de42944dffaeb3ef5548bff3a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:43.593086 13661 raft_consensus.cc:1261] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 1 FOLLOWER]: Refusing update from remote peer 27cfd0de42944dffaeb3ef5548bff3a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:43.593926 13869 consensus_queue.cc:1035] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a6d1a2de7844459b195943560fadf2" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:43.594274 13863 consensus_queue.cc:1035] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3f6823b0644a36811ceb835e718ac3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 01:02:43.599346 13707 tablet.cc:2369] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:02:43.607156 13872 mvcc.cc:205] Tried to move back new op lower bound from 7332707383667671040 to 7332707383556026368. Current Snapshot: MvccSnapshot[applied={T|T < 7332707383667671040}]
I20260924 01:02:43.607298 13873 mvcc.cc:205] Tried to move back new op lower bound from 7332707383667671040 to 7332707383556026368. Current Snapshot: MvccSnapshot[applied={T|T < 7332707383667671040}]
I20260924 01:02:43.607450 13874 mvcc.cc:205] Tried to move back new op lower bound from 7332707383667671040 to 7332707383556026368. Current Snapshot: MvccSnapshot[applied={T|T < 7332707383667671040}]
I20260924 01:02:43.621558 13384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:43.621930 13384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:43.625932 13384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66544e187884521ac2678cba8df88a2: Sending DeleteTablet for 3 replicas of tablet 194512549c9d4496ae508a6cc0cafe21
I20260924 01:02:43.627229 13508 tablet_service.cc:1514] Processing DeleteTablet for tablet 194512549c9d4496ae508a6cc0cafe21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:43 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:54482
I20260924 01:02:43.627368 13773 tablet_service.cc:1514] Processing DeleteTablet for tablet 194512549c9d4496ae508a6cc0cafe21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:43 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:59864
I20260924 01:02:43.627424 13641 tablet_service.cc:1514] Processing DeleteTablet for tablet 194512549c9d4496ae508a6cc0cafe21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:43 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:46142
I20260924 01:02:43.630002 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13446
I20260924 01:02:43.638444 13881 tablet_replica.cc:330] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: stopping tablet replica
I20260924 01:02:43.638736 13883 tablet_replica.cc:330] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: stopping tablet replica
I20260924 01:02:43.638955 13882 tablet_replica.cc:330] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: stopping tablet replica
I20260924 01:02:43.639235 13881 raft_consensus.cc:2229] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:43.639371 13882 raft_consensus.cc:2229] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:43.639375 13883 raft_consensus.cc:2229] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:43.639915 13883 raft_consensus.cc:2258] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:43.639915 13882 raft_consensus.cc:2258] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:43.640208 13881 raft_consensus.cc:2258] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:43.642426 13883 ts_tablet_manager.cc:1905] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:43.642426 13882 ts_tablet_manager.cc:1905] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:43.643895 13881 ts_tablet_manager.cc:1905] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:43.649588 13882 ts_tablet_manager.cc:1918] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:43.649873 13882 log.cc:1196] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/194512549c9d4496ae508a6cc0cafe21
I20260924 01:02:43.649902 13883 ts_tablet_manager.cc:1918] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:43.650089 13883 log.cc:1196] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/194512549c9d4496ae508a6cc0cafe21
I20260924 01:02:43.650563 13882 ts_tablet_manager.cc:1939] T 194512549c9d4496ae508a6cc0cafe21 P c2a6d1a2de7844459b195943560fadf2: Deleting consensus metadata
I20260924 01:02:43.650674 13883 ts_tablet_manager.cc:1939] T 194512549c9d4496ae508a6cc0cafe21 P bd3f6823b0644a36811ceb835e718ac3: Deleting consensus metadata
I20260924 01:02:43.651247 13881 ts_tablet_manager.cc:1918] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 24 01:02:43 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:43.652681 13370 catalog_manager.cc:4841] TS c2a6d1a2de7844459b195943560fadf2 (127.8.254.67:44283): tablet 194512549c9d4496ae508a6cc0cafe21 (table test-table [id=1520038f8cfa452787648bbb3fc15e95]) successfully deleted
I20260924 01:02:43.653002 13370 catalog_manager.cc:4841] TS bd3f6823b0644a36811ceb835e718ac3 (127.8.254.66:36787): tablet 194512549c9d4496ae508a6cc0cafe21 (table test-table [id=1520038f8cfa452787648bbb3fc15e95]) successfully deleted
I20260924 01:02:43.653427 13881 log.cc:1196] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/194512549c9d4496ae508a6cc0cafe21
I20260924 01:02:43.654028 13881 ts_tablet_manager.cc:1939] T 194512549c9d4496ae508a6cc0cafe21 P 27cfd0de42944dffaeb3ef5548bff3a5: Deleting consensus metadata
I20260924 01:02:43.656168 13369 catalog_manager.cc:4841] TS 27cfd0de42944dffaeb3ef5548bff3a5 (127.8.254.65:41357): tablet 194512549c9d4496ae508a6cc0cafe21 (table test-table [id=1520038f8cfa452787648bbb3fc15e95]) successfully deleted
I20260924 01:02:43.660907 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.632136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.8.254.65:41357)
0924 01:02:43.632423 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.632433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.641565 (+ 9132us) server_negotiation.cc:407] Connection header received
0924 01:02:43.641872 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.641877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.641968 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.642051 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:43.643940 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.644963 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.646884 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.647585 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.647692 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.654404 (+ 6712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:43.654418 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:43.654435 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:43.654487 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:43.657972 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.658917 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.658924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.658930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.659076 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.659504 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.659507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.659509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.659876 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:43.660100 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.660448 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.660623 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 01:02:43.661610 13568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:43.767514 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13446
I20260924 01:02:43.786418 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13579
Sep 24 01:02:43 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:43.801492 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.787234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.8.254.66:36787)
0924 01:02:43.787441 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.787445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.787463 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:02:43.787612 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.787616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.787680 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.787781 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:43.788963 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.789696 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.791107 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.791494 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.791593 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.795664 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:43.795700 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:43.795716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:43.795763 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:43.798901 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.799621 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.799629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.799632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.799772 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.800124 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.800128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.800136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.800555 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:43.800778 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.801035 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.801210 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 01:02:43.802307 13701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:43.893947 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13579
I20260924 01:02:43.914551 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13711
Sep 24 01:02:43 dist-test-slave-5c7l krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:43.929354 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:43.915331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.8.254.67:44283)
0924 01:02:43.915501 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:43.915506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:43.915555 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:02:43.915720 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:43.915724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:43.915796 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:43.915894 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:43.917083 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.917757 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.918985 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:43.919387 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:43.919480 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:43.924113 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:43.924128 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:43.924145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:43.924178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:43.927053 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.927663 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.927671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.927673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.927793 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:43.928097 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:43.928100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:43.928102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:43.928455 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:43.928667 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:43.928921 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 01:02:43.929079 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:02:43.930024 13833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:44.022506 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13711
I20260924 01:02:44.039485 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13354
I20260924 01:02:44.040709 13415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:44.148787 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13354
2026-09-24T01:02:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4119 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T01:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:44Z Disabled control of system clock
I20260924 01:02:44.216621 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:46477
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 01:02:44.545753 13892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:44.546056 13892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:44.546141 13892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:44.553454 13892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:44.553612 13892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:44.553686 13892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:44.553772 13892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:44.553818 13892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:44.563344 13892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:46477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:44.564743 13892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:44.566550 13892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:44.578657 13897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:44.578609 13898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:44.578553 13900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:44.579289 13892 server_base.cc:1034] running on GCE node
I20260924 01:02:44.580070 13892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:44.581580 13892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:44.582814 13892 hybrid_clock.cc:648] HybridClock initialized: now 1790211764582769 us; error 40 us; skew 500 ppm
I20260924 01:02:44.583184 13892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:44.585618 13892 webserver.cc:460] Webserver started at http://127.8.254.126:45007/ using document root <none> and password file <none>
I20260924 01:02:44.586426 13892 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:44.586560 13892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:44.586982 13892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:44.589946 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "b996d44ec83c45d6bd530812da92a9ac"
format_stamp: "Formatted at 2026-09-24 01:02:44 on dist-test-slave-5c7l"
server_key: "74252a6ec7a28d952bdcf5e57f7a676d"
server_key_iv: "8686025bb2e52677ed2a939978ce17ca"
server_key_version: "encryptionkey@0"
I20260924 01:02:44.590855 13892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "b996d44ec83c45d6bd530812da92a9ac"
format_stamp: "Formatted at 2026-09-24 01:02:44 on dist-test-slave-5c7l"
server_key: "74252a6ec7a28d952bdcf5e57f7a676d"
server_key_iv: "8686025bb2e52677ed2a939978ce17ca"
server_key_version: "encryptionkey@0"
I20260924 01:02:44.596043 13892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 01:02:44.599499 13906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:44.600881 13892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:44.601084 13892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "b996d44ec83c45d6bd530812da92a9ac"
format_stamp: "Formatted at 2026-09-24 01:02:44 on dist-test-slave-5c7l"
server_key: "74252a6ec7a28d952bdcf5e57f7a676d"
server_key_iv: "8686025bb2e52677ed2a939978ce17ca"
server_key_version: "encryptionkey@0"
I20260924 01:02:44.601336 13892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:44.607111 13892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:44.607921 13892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:44.608238 13892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:44.626482 13892 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:46477
I20260924 01:02:44.626516 13957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:46477 every 8 connection(s)
I20260924 01:02:44.628049 13892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:44.631944 13958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:44.635298 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13892
I20260924 01:02:44.635522 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:44.636168 9209 external_mini_cluster.cc:1428] Setting key 5e0f0044ed88a7bf01f6dfcf55504d47
I20260924 01:02:44.644014 13958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac: Bootstrap starting.
I20260924 01:02:44.646090 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:44.638288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.8.254.126:46477)
0924 01:02:44.639411 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:44.639422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:44.639465 (+ 43us) server_negotiation.cc:407] Connection header received
0924 01:02:44.642331 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:44.642359 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:44.642809 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:44.643248 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:44.644363 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:44.644370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:44.644385 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:44.644399 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:44.644460 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:44.644476 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:44.644732 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:44.644880 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 01:02:44.645143 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":359,"threads_started":1}
I20260924 01:02:44.647774 13958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:44.648912 13958 log.cc:824] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:44.651737 13958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac: No bootstrap required, opened a new log
I20260924 01:02:44.659438 13958 raft_consensus.cc:348] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b996d44ec83c45d6bd530812da92a9ac" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46477 } }
I20260924 01:02:44.659789 13958 raft_consensus.cc:374] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:44.659864 13958 raft_consensus.cc:729] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b996d44ec83c45d6bd530812da92a9ac, State: Initialized, Role: FOLLOWER
I20260924 01:02:44.660439 13958 consensus_queue.cc:260] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b996d44ec83c45d6bd530812da92a9ac" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46477 } }
I20260924 01:02:44.660650 13958 raft_consensus.cc:388] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:44.660748 13958 raft_consensus.cc:482] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:44.660874 13958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:44.663060 13958 raft_consensus.cc:504] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b996d44ec83c45d6bd530812da92a9ac" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46477 } }
I20260924 01:02:44.663625 13958 leader_election.cc:302] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b996d44ec83c45d6bd530812da92a9ac; no voters:
I20260924 01:02:44.664369 13958 leader_election.cc:288] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:44.664520 13963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:44.665261 13963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [term 1 LEADER]: Becoming Leader. State: Replica: b996d44ec83c45d6bd530812da92a9ac, State: Running, Role: LEADER
I20260924 01:02:44.665797 13963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b996d44ec83c45d6bd530812da92a9ac" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46477 } }
I20260924 01:02:44.666592 13958 sys_catalog.cc:564] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:44.668785 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b996d44ec83c45d6bd530812da92a9ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b996d44ec83c45d6bd530812da92a9ac" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46477 } } }
I20260924 01:02:44.669204 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:44.668766 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b996d44ec83c45d6bd530812da92a9ac. Latest consensus state: current_term: 1 leader_uuid: "b996d44ec83c45d6bd530812da92a9ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b996d44ec83c45d6bd530812da92a9ac" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46477 } } }
I20260924 01:02:44.669348 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:44.679780 13969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:44.684340 13969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:44.693301 13969 catalog_manager.cc:1347] Generated new cluster ID: 75ed4a92d4e04e17ab72e41257616ef8
I20260924 01:02:44.693425 13969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:44.715605 13969 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:44.716192 13969 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:44.738706 13969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac: Generated new TSK 0
I20260924 01:02:44.739638 13969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:02:44.746047 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46477
--builtin_ntp_servers=127.8.254.84:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 01:02:45.072295 13982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:45.072593 13982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:45.072685 13982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:45.080814 13982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:45.080938 13982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:45.081053 13982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:45.090659 13982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:45.091954 13982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:45.093698 13982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:45.107306 13987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:45.107225 13990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:45.107335 13988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:45.108598 13982 server_base.cc:1034] running on GCE node
I20260924 01:02:45.109225 13982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:45.110249 13982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:45.111521 13982 hybrid_clock.cc:648] HybridClock initialized: now 1790211765111465 us; error 54 us; skew 500 ppm
I20260924 01:02:45.111878 13982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:45.118911 13982 webserver.cc:460] Webserver started at http://127.8.254.65:41367/ using document root <none> and password file <none>
I20260924 01:02:45.119670 13982 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:45.119817 13982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:45.120157 13982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:45.123294 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "08c46bdf4dcd4350ac16be278a2b17dd"
format_stamp: "Formatted at 2026-09-24 01:02:45 on dist-test-slave-5c7l"
server_key: "78e0f1ccdac716430b32889eedca7d8b"
server_key_iv: "60470f75c789f35322195cc0b592479a"
server_key_version: "encryptionkey@0"
I20260924 01:02:45.124157 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "08c46bdf4dcd4350ac16be278a2b17dd"
format_stamp: "Formatted at 2026-09-24 01:02:45 on dist-test-slave-5c7l"
server_key: "78e0f1ccdac716430b32889eedca7d8b"
server_key_iv: "60470f75c789f35322195cc0b592479a"
server_key_version: "encryptionkey@0"
I20260924 01:02:45.129534 13982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 01:02:45.133121 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:45.134539 13982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:45.134722 13982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "08c46bdf4dcd4350ac16be278a2b17dd"
format_stamp: "Formatted at 2026-09-24 01:02:45 on dist-test-slave-5c7l"
server_key: "78e0f1ccdac716430b32889eedca7d8b"
server_key_iv: "60470f75c789f35322195cc0b592479a"
server_key_version: "encryptionkey@0"
I20260924 01:02:45.134975 13982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:45.140866 13982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:45.141732 13982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:45.142031 13982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:45.143049 13982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:45.144462 13982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:45.144615 13982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:45.144750 13982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:45.144850 13982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:45.179728 13982 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:39321
I20260924 01:02:45.179759 14108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:39321 every 8 connection(s)
I20260924 01:02:45.181139 13982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:45.185858 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 13982
I20260924 01:02:45.186096 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:45.186769 9209 external_mini_cluster.cc:1428] Setting key 52cadbe6f0ed3c692118a2b4c7e057a1
I20260924 01:02:45.189484 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46477
--builtin_ntp_servers=127.8.254.84:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 01:02:45.189963 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:45.183785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:42337 (local address 127.8.254.126:46477)
0924 01:02:45.184206 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:45.184211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:45.185569 (+ 1358us) server_negotiation.cc:407] Connection header received
0924 01:02:45.187483 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:45.187488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:45.187568 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:45.187695 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:45.189121 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:45.189128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:45.189133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:45.189157 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:45.189198 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:45.189210 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:45.189339 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:45.189605 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 01:02:45.189674 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":214,"threads_started":1}
I20260924 01:02:45.191022 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:45.184414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:46477 (local address 127.8.254.65:42337)
0924 01:02:45.185411 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:45.185450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:45.187205 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:45.187991 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:45.187999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:45.188129 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:45.188824 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:45.188827 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:45.188865 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:45.188953 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:45.189305 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:45.189318 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:45.189326 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 01:02:45.189640 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 01:02:45.189905 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":254,"threads_started":1}
I20260924 01:02:45.192739 14109 heartbeater.cc:347] Connected to a master server at 127.8.254.126:46477
I20260924 01:02:45.193061 14109 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:45.193722 14109 heartbeater.cc:511] Master 127.8.254.126:46477 requested a full tablet report, sending...
I20260924 01:02:45.196291 13923 ts_manager.cc:194] Registered new tserver with Master: 08c46bdf4dcd4350ac16be278a2b17dd (127.8.254.65:39321)
W20260924 01:02:45.523716 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:45.524065 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:45.524287 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:45.531857 14113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:45.531971 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:45.532143 14113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:45.541986 14113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:45.543771 14113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:45.545542 14113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:45.558800 14121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:45.558862 14119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:45.558881 14118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:45.559737 14113 server_base.cc:1034] running on GCE node
I20260924 01:02:45.560434 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:45.561472 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:45.562707 14113 hybrid_clock.cc:648] HybridClock initialized: now 1790211765562656 us; error 63 us; skew 500 ppm
I20260924 01:02:45.563057 14113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:45.565507 14113 webserver.cc:460] Webserver started at http://127.8.254.66:33091/ using document root <none> and password file <none>
I20260924 01:02:45.566340 14113 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:45.566496 14113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:45.566847 14113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:45.569990 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "ed9505c524d141199fe0b1ffdfe115b5"
format_stamp: "Formatted at 2026-09-24 01:02:45 on dist-test-slave-5c7l"
server_key: "e27d4750cf96d900c61f49a0590aab61"
server_key_iv: "5436c4d690c7e48fce16d59c30fcf495"
server_key_version: "encryptionkey@0"
I20260924 01:02:45.570835 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "ed9505c524d141199fe0b1ffdfe115b5"
format_stamp: "Formatted at 2026-09-24 01:02:45 on dist-test-slave-5c7l"
server_key: "e27d4750cf96d900c61f49a0590aab61"
server_key_iv: "5436c4d690c7e48fce16d59c30fcf495"
server_key_version: "encryptionkey@0"
I20260924 01:02:45.576288 14113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:02:45.579896 14127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:45.581267 14113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:02:45.581463 14113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "ed9505c524d141199fe0b1ffdfe115b5"
format_stamp: "Formatted at 2026-09-24 01:02:45 on dist-test-slave-5c7l"
server_key: "e27d4750cf96d900c61f49a0590aab61"
server_key_iv: "5436c4d690c7e48fce16d59c30fcf495"
server_key_version: "encryptionkey@0"
I20260924 01:02:45.581698 14113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:45.588109 14113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:45.589015 14113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:45.589288 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:45.590281 14113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:45.591727 14113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:45.591814 14113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:45.591962 14113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:45.592052 14113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:45.625756 14113 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:38589
I20260924 01:02:45.625768 14239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:38589 every 8 connection(s)
I20260924 01:02:45.627454 14113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:45.630854 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14113
I20260924 01:02:45.631054 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:45.631669 9209 external_mini_cluster.cc:1428] Setting key c8576d7ae5bcf32aec35638a7320814b
I20260924 01:02:45.635025 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46477
--builtin_ntp_servers=127.8.254.84:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 01:02:45.637079 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:45.630386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:43081 (local address 127.8.254.126:46477)
0924 01:02:45.630609 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:45.630615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:45.632301 (+ 1686us) server_negotiation.cc:407] Connection header received
0924 01:02:45.634444 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:45.634451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:45.634511 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:45.634616 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:45.636248 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:45.636255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:45.636260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:45.636281 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:45.636326 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:45.636341 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:45.636468 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:45.636728 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 01:02:45.636792 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 01:02:45.638736 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:45.630927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:46477 (local address 127.8.254.66:43081)
0924 01:02:45.632090 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:45.632139 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:45.634106 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:45.634911 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:45.634924 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:45.635059 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:45.635854 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:45.635860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:45.635926 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:45.636055 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:45.636464 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:45.636477 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:45.636490 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 01:02:45.636852 (+ 362us) client_negotiation.cc:240] Negotiation successful
0924 01:02:45.637138 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":339,"threads_started":1}
I20260924 01:02:45.640316 14240 heartbeater.cc:347] Connected to a master server at 127.8.254.126:46477
I20260924 01:02:45.640677 14240 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:45.641399 14240 heartbeater.cc:511] Master 127.8.254.126:46477 requested a full tablet report, sending...
I20260924 01:02:45.642997 13923 ts_manager.cc:194] Registered new tserver with Master: ed9505c524d141199fe0b1ffdfe115b5 (127.8.254.66:38589)
W20260924 01:02:45.971778 14243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:45.972087 14243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:45.972261 14243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:45.979840 14243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:45.979950 14243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:45.980129 14243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:45.989970 14243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:45.991335 14243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:45.993497 14243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:46.007706 14249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:46.007840 14248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:46.007762 14251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:46.008543 14243 server_base.cc:1034] running on GCE node
I20260924 01:02:46.009239 14243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:46.010267 14243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:46.011499 14243 hybrid_clock.cc:648] HybridClock initialized: now 1790211766011449 us; error 41 us; skew 500 ppm
I20260924 01:02:46.011845 14243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:46.014180 14243 webserver.cc:460] Webserver started at http://127.8.254.67:36821/ using document root <none> and password file <none>
I20260924 01:02:46.014948 14243 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:46.015082 14243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:46.015417 14243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:46.018392 14243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "cef37950d5ec40bb8ea16b0647b0270b"
format_stamp: "Formatted at 2026-09-24 01:02:46 on dist-test-slave-5c7l"
server_key: "66e880820441e3cc0d4da971521714d4"
server_key_iv: "ea50d672b2d53bf6b2dd3184c4ca2459"
server_key_version: "encryptionkey@0"
I20260924 01:02:46.019192 14243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "cef37950d5ec40bb8ea16b0647b0270b"
format_stamp: "Formatted at 2026-09-24 01:02:46 on dist-test-slave-5c7l"
server_key: "66e880820441e3cc0d4da971521714d4"
server_key_iv: "ea50d672b2d53bf6b2dd3184c4ca2459"
server_key_version: "encryptionkey@0"
I20260924 01:02:46.024446 14243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 01:02:46.028009 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:46.029361 14243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:02:46.029565 14243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "cef37950d5ec40bb8ea16b0647b0270b"
format_stamp: "Formatted at 2026-09-24 01:02:46 on dist-test-slave-5c7l"
server_key: "66e880820441e3cc0d4da971521714d4"
server_key_iv: "ea50d672b2d53bf6b2dd3184c4ca2459"
server_key_version: "encryptionkey@0"
I20260924 01:02:46.029815 14243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:46.035676 14243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:46.036475 14243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:46.036744 14243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:46.037636 14243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:46.039260 14243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:46.039392 14243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:46.039520 14243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:46.039605 14243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:46.072006 14243 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:35177
I20260924 01:02:46.072057 14369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:35177 every 8 connection(s)
I20260924 01:02:46.073412 14243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:46.078967 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14243
I20260924 01:02:46.079187 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:46.079622 9209 external_mini_cluster.cc:1428] Setting key 4cc2aaa82e6bc9e62767835b783d3efe
I20260924 01:02:46.081971 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.076129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:53093 (local address 127.8.254.126:46477)
0924 01:02:46.076309 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.076314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.077672 (+ 1358us) server_negotiation.cc:407] Connection header received
0924 01:02:46.079502 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.079509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.079589 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.079702 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.081212 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.081225 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.081229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.081245 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.081278 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.081289 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.081412 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.081619 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.081686 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:02:46.083019 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.076668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:46477 (local address 127.8.254.67:53093)
0924 01:02:46.077544 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.077585 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:46.079195 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:46.079936 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:46.079944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:46.080033 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:46.080823 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:46.080826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:46.080910 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:46.081041 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:46.081394 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:46.081404 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:46.081411 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 01:02:46.081699 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 01:02:46.081956 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260924 01:02:46.084307 14370 heartbeater.cc:347] Connected to a master server at 127.8.254.126:46477
I20260924 01:02:46.084595 14370 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:46.085202 14370 heartbeater.cc:511] Master 127.8.254.126:46477 requested a full tablet report, sending...
I20260924 01:02:46.086622 13923 ts_manager.cc:194] Registered new tserver with Master: cef37950d5ec40bb8ea16b0647b0270b (127.8.254.67:35177)
I20260924 01:02:46.094959 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:02:46.101547 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.099360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.8.254.126:46477)
0924 01:02:46.099520 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.099524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.099972 (+ 448us) server_negotiation.cc:407] Connection header received
0924 01:02:46.100103 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.100106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.100160 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.100239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.101059 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.101063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.101066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.101081 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.101111 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.101123 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.101219 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.101302 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.101347 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 01:02:46.105611 13923 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:02:46.112383 13923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:02:46.129319 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.124047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:35177 (local address 127.0.0.1:60576)
0924 01:02:46.125065 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.125115 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:46.125276 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:46.126226 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:46.126233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:46.126338 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:46.127118 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:46.127121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:46.127149 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:46.127237 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:46.128515 (+ 1278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:46.128529 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:46.128539 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 01:02:46.128720 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 01:02:46.128797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":460,"threads_started":1}
I20260924 01:02:46.130102 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.124245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.8.254.67:35177)
0924 01:02:46.125624 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.125633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.125669 (+ 36us) server_negotiation.cc:407] Connection header received
0924 01:02:46.125741 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.125749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.125998 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.126165 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.128252 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.128258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.128269 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.128281 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.128324 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.128373 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.129252 (+ 879us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.129331 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.129692 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":1000,"threads_started":1}
I20260924 01:02:46.130442 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.125320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46856)
0924 01:02:46.125777 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.125790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:46.125890 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:46.128032 (+ 2142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:46.128037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:46.128060 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:46.128972 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:46.128978 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:46.129026 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:46.129084 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:46.129787 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:46.129794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:46.129801 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 01:02:46.129941 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 01:02:46.129993 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":558,"threads_started":1}
I20260924 01:02:46.130817 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.126234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:38589 (local address 127.0.0.1:38690)
0924 01:02:46.127423 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.127438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:46.127528 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:46.128511 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:46.128516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:46.128537 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:46.129165 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:46.129168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:46.129183 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:46.129215 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:46.129855 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:46.129863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:46.129871 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 01:02:46.130042 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 01:02:46.130098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":881,"threads_started":1}
I20260924 01:02:46.131340 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.126780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.8.254.66:38589)
0924 01:02:46.127992 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.128000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.128023 (+ 23us) server_negotiation.cc:407] Connection header received
0924 01:02:46.128085 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.128095 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.128352 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.128516 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.129616 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.129626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.129640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.129672 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.129718 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.129740 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.129911 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.130022 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.130116 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":924,"threads_started":1}
I20260924 01:02:46.131388 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.125743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.8.254.65:39321)
0924 01:02:46.126263 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.126271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.126297 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:02:46.126360 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.126367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.126594 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.127151 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.129467 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.129476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.129492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.129513 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.129571 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.129595 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.129719 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.130481 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.131088 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":1208,"threads_started":1}
I20260924 01:02:46.134450 14305 tablet_service.cc:1467] Processing CreateTablet for tablet e3305374c94c4d92aafa1c85a8767958 (DEFAULT_TABLE table=test-table [id=a8000a37df3d4335bf987d971978fb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:46.134640 14175 tablet_service.cc:1467] Processing CreateTablet for tablet e3305374c94c4d92aafa1c85a8767958 (DEFAULT_TABLE table=test-table [id=a8000a37df3d4335bf987d971978fb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:46.135017 14044 tablet_service.cc:1467] Processing CreateTablet for tablet e3305374c94c4d92aafa1c85a8767958 (DEFAULT_TABLE table=test-table [id=a8000a37df3d4335bf987d971978fb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:02:46.135952 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3305374c94c4d92aafa1c85a8767958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:46.136047 14175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3305374c94c4d92aafa1c85a8767958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:46.136186 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3305374c94c4d92aafa1c85a8767958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:46.147950 14388 tablet_bootstrap.cc:492] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Bootstrap starting.
I20260924 01:02:46.150362 14389 tablet_bootstrap.cc:492] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Bootstrap starting.
I20260924 01:02:46.151789 14388 tablet_bootstrap.cc:654] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:46.152136 14390 tablet_bootstrap.cc:492] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Bootstrap starting.
I20260924 01:02:46.153291 14388 log.cc:824] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:46.154109 14389 tablet_bootstrap.cc:654] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:46.155519 14390 tablet_bootstrap.cc:654] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:46.155722 14389 log.cc:824] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:46.156605 14390 log.cc:824] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:46.159075 14389 tablet_bootstrap.cc:492] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: No bootstrap required, opened a new log
I20260924 01:02:46.159363 14389 ts_tablet_manager.cc:1397] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 01:02:46.159709 14388 tablet_bootstrap.cc:492] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: No bootstrap required, opened a new log
I20260924 01:02:46.159711 14390 tablet_bootstrap.cc:492] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: No bootstrap required, opened a new log
I20260924 01:02:46.159974 14388 ts_tablet_manager.cc:1397] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 01:02:46.159982 14390 ts_tablet_manager.cc:1397] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 01:02:46.167073 14389 raft_consensus.cc:348] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.167481 14389 raft_consensus.cc:374] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:46.167573 14389 raft_consensus.cc:729] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c46bdf4dcd4350ac16be278a2b17dd, State: Initialized, Role: FOLLOWER
I20260924 01:02:46.168251 14389 consensus_queue.cc:260] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.169378 14109 heartbeater.cc:503] Master 127.8.254.126:46477 was elected leader, sending a full tablet report...
I20260924 01:02:46.170130 14389 ts_tablet_manager.cc:1428] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260924 01:02:46.170661 14388 raft_consensus.cc:348] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.170661 14390 raft_consensus.cc:348] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.171106 14390 raft_consensus.cc:374] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:46.171106 14388 raft_consensus.cc:374] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:46.171212 14390 raft_consensus.cc:729] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef37950d5ec40bb8ea16b0647b0270b, State: Initialized, Role: FOLLOWER
I20260924 01:02:46.171212 14388 raft_consensus.cc:729] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9505c524d141199fe0b1ffdfe115b5, State: Initialized, Role: FOLLOWER
I20260924 01:02:46.171835 14390 consensus_queue.cc:260] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.171825 14388 consensus_queue.cc:260] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.172989 14370 heartbeater.cc:503] Master 127.8.254.126:46477 was elected leader, sending a full tablet report...
I20260924 01:02:46.173987 14390 ts_tablet_manager.cc:1428] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 01:02:46.175642 14388 ts_tablet_manager.cc:1428] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260924 01:02:46.175860 14240 heartbeater.cc:503] Master 127.8.254.126:46477 was elected leader, sending a full tablet report...
W20260924 01:02:46.183517 14110 tablet.cc:2369] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:46.325186 14371 tablet.cc:2369] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:02:46.379587 14241 tablet.cc:2369] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:02:46.594574 14394 raft_consensus.cc:482] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:02:46.594933 14394 raft_consensus.cc:504] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.596764 14394 leader_election.cc:288] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef37950d5ec40bb8ea16b0647b0270b (127.8.254.67:35177), ed9505c524d141199fe0b1ffdfe115b5 (127.8.254.66:38589)
I20260924 01:02:46.600287 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.597167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:40715 (local address 127.8.254.67:35177)
0924 01:02:46.597417 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.597445 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.598006 (+ 561us) server_negotiation.cc:407] Connection header received
0924 01:02:46.598174 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.598182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.598327 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.598452 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.599608 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.599612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.599615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.599630 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.599658 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.599670 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.599807 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.599944 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.600009 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 01:02:46.600456 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.597170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:35177 (local address 127.8.254.65:40715)
0924 01:02:46.597843 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.597863 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:46.598031 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:46.598500 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:46.598506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:46.598537 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:46.599347 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:46.599352 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:46.599375 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:46.599454 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:46.599794 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:46.599799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:46.599804 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 01:02:46.599986 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 01:02:46.600038 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":367,"threads_started":1}
I20260924 01:02:46.600834 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.598139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:54157 (local address 127.8.254.66:38589)
0924 01:02:46.598335 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.598344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.598871 (+ 527us) server_negotiation.cc:407] Connection header received
0924 01:02:46.599042 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.599048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.599110 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.599211 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.600156 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.600162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.600166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.600184 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.600221 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.600233 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.600356 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.600496 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.600585 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:02:46.601382 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3305374c94c4d92aafa1c85a8767958" candidate_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef37950d5ec40bb8ea16b0647b0270b" is_pre_election: true
I20260924 01:02:46.601886 14325 raft_consensus.cc:2454] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c46bdf4dcd4350ac16be278a2b17dd in term 0.
I20260924 01:02:46.602289 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.598006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:38589 (local address 127.8.254.65:54157)
0924 01:02:46.598757 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.598779 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:46.598909 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:46.599268 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:46.599274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:46.599307 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:46.599958 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:02:46.599961 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:46.599977 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:02:46.600011 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:46.600327 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:46.600334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:46.600340 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 01:02:46.601758 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0924 01:02:46.601820 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":387,"threads_started":1}
I20260924 01:02:46.602586 13999 leader_election.cc:302] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c46bdf4dcd4350ac16be278a2b17dd, cef37950d5ec40bb8ea16b0647b0270b; no voters:
I20260924 01:02:46.602977 14394 raft_consensus.cc:2781] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:02:46.603154 14394 raft_consensus.cc:482] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:02:46.603269 14394 raft_consensus.cc:3037] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:46.603790 14195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3305374c94c4d92aafa1c85a8767958" candidate_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9505c524d141199fe0b1ffdfe115b5" is_pre_election: true
I20260924 01:02:46.604321 14195 raft_consensus.cc:2454] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c46bdf4dcd4350ac16be278a2b17dd in term 0.
I20260924 01:02:46.606024 14394 raft_consensus.cc:504] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.606592 14394 leader_election.cc:288] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [CANDIDATE]: Term 1 election: Requested vote from peers cef37950d5ec40bb8ea16b0647b0270b (127.8.254.67:35177), ed9505c524d141199fe0b1ffdfe115b5 (127.8.254.66:38589)
I20260924 01:02:46.607077 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3305374c94c4d92aafa1c85a8767958" candidate_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef37950d5ec40bb8ea16b0647b0270b"
I20260924 01:02:46.607139 14195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3305374c94c4d92aafa1c85a8767958" candidate_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9505c524d141199fe0b1ffdfe115b5"
I20260924 01:02:46.607275 14325 raft_consensus.cc:3037] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:46.607336 14195 raft_consensus.cc:3037] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:46.609404 14325 raft_consensus.cc:2454] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c46bdf4dcd4350ac16be278a2b17dd in term 1.
I20260924 01:02:46.609469 14195 raft_consensus.cc:2454] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c46bdf4dcd4350ac16be278a2b17dd in term 1.
I20260924 01:02:46.609809 13999 leader_election.cc:302] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c46bdf4dcd4350ac16be278a2b17dd, cef37950d5ec40bb8ea16b0647b0270b; no voters:
I20260924 01:02:46.610105 14394 raft_consensus.cc:2781] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:46.610569 14394 raft_consensus.cc:686] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 1 LEADER]: Becoming Leader. State: Replica: 08c46bdf4dcd4350ac16be278a2b17dd, State: Running, Role: LEADER
I20260924 01:02:46.611033 14394 consensus_queue.cc:237] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } }
I20260924 01:02:46.615806 13922 catalog_manager.cc:5495] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 08c46bdf4dcd4350ac16be278a2b17dd (127.8.254.65). New cstate: current_term: 1 leader_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08c46bdf4dcd4350ac16be278a2b17dd" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20260924 01:02:46.698249 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.695513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.8.254.65:39321)
0924 01:02:46.696241 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.696246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.696263 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:02:46.696361 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.696365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.696443 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.696536 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.697533 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.697537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.697540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.697554 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.697593 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.697604 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.697699 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.697845 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.697893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":517,"threads_started":1}
I20260924 01:02:46.705824 14195 raft_consensus.cc:1261] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 1 FOLLOWER]: Refusing update from remote peer 08c46bdf4dcd4350ac16be278a2b17dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:46.706086 14325 raft_consensus.cc:1261] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 1 FOLLOWER]: Refusing update from remote peer 08c46bdf4dcd4350ac16be278a2b17dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:02:46.706799 14399 consensus_queue.cc:1035] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed9505c524d141199fe0b1ffdfe115b5" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:46.707194 14394 consensus_queue.cc:1035] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef37950d5ec40bb8ea16b0647b0270b" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:46.718716 14408 mvcc.cc:205] Tried to move back new op lower bound from 7332707396417482752 to 7332707396042424320. Current Snapshot: MvccSnapshot[applied={T|T < 7332707396417482752}]
I20260924 01:02:46.736568 13922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:46.736858 13922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:02:46.740765 13922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b996d44ec83c45d6bd530812da92a9ac: Sending DeleteTablet for 3 replicas of tablet e3305374c94c4d92aafa1c85a8767958
I20260924 01:02:46.742072 14305 tablet_service.cc:1514] Processing DeleteTablet for tablet e3305374c94c4d92aafa1c85a8767958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:46 UTC) from {username='slave'} at 127.0.0.1:60576
I20260924 01:02:46.742290 14175 tablet_service.cc:1514] Processing DeleteTablet for tablet e3305374c94c4d92aafa1c85a8767958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:46 UTC) from {username='slave'} at 127.0.0.1:38690
I20260924 01:02:46.743052 14416 tablet_replica.cc:330] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: stopping tablet replica
I20260924 01:02:46.742973 14044 tablet_service.cc:1514] Processing DeleteTablet for tablet e3305374c94c4d92aafa1c85a8767958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:02:46 UTC) from {username='slave'} at 127.0.0.1:46856
I20260924 01:02:46.743387 14416 raft_consensus.cc:2229] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:46.743855 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13982
I20260924 01:02:46.743984 14416 raft_consensus.cc:2258] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:46.748041 14418 tablet_replica.cc:330] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: stopping tablet replica
I20260924 01:02:46.750025 14416 ts_tablet_manager.cc:1905] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:46.754668 14418 raft_consensus.cc:2229] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:02:46.755149 14417 tablet_replica.cc:330] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: stopping tablet replica
I20260924 01:02:46.755405 14418 raft_consensus.cc:2258] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:46.755609 14417 raft_consensus.cc:2229] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:02:46.756500 14417 raft_consensus.cc:2258] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:02:46.757624 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.744658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.8.254.65:39321)
0924 01:02:46.744833 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.744839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.754835 (+ 9996us) server_negotiation.cc:407] Connection header received
0924 01:02:46.755130 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.755137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.755213 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.755337 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.756777 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.756784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.756789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.756809 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.756849 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.756872 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.756976 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.757214 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.757297 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 01:02:46.757403 14418 ts_tablet_manager.cc:1905] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:46.758419 14104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:46.758457 14417 ts_tablet_manager.cc:1905] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:46.762827 14416 ts_tablet_manager.cc:1918] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:46.763010 14416 log.cc:1196] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/e3305374c94c4d92aafa1c85a8767958
I20260924 01:02:46.763525 14416 ts_tablet_manager.cc:1939] T e3305374c94c4d92aafa1c85a8767958 P cef37950d5ec40bb8ea16b0647b0270b: Deleting consensus metadata
I20260924 01:02:46.765599 14417 ts_tablet_manager.cc:1918] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:02:46.765764 14417 log.cc:1196] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/e3305374c94c4d92aafa1c85a8767958
I20260924 01:02:46.765978 13909 catalog_manager.cc:4841] TS cef37950d5ec40bb8ea16b0647b0270b (127.8.254.67:35177): tablet e3305374c94c4d92aafa1c85a8767958 (table test-table [id=a8000a37df3d4335bf987d971978fb68]) successfully deleted
I20260924 01:02:46.766328 14417 ts_tablet_manager.cc:1939] T e3305374c94c4d92aafa1c85a8767958 P ed9505c524d141199fe0b1ffdfe115b5: Deleting consensus metadata
I20260924 01:02:46.768024 13909 catalog_manager.cc:4841] TS ed9505c524d141199fe0b1ffdfe115b5 (127.8.254.66:38589): tablet e3305374c94c4d92aafa1c85a8767958 (table test-table [id=a8000a37df3d4335bf987d971978fb68]) successfully deleted
I20260924 01:02:46.862390 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13982
I20260924 01:02:46.862494 14418 ts_tablet_manager.cc:1918] T e3305374c94c4d92aafa1c85a8767958 P 08c46bdf4dcd4350ac16be278a2b17dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 01:02:46.880203 13909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.254.65:39321 (error 108)
I20260924 01:02:46.880896 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14113
I20260924 01:02:46.882536 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.881575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46884)
0924 01:02:46.882161 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.882304 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":309,"threads_started":1}
W20260924 01:02:46.882781 13909 catalog_manager.cc:4584] TS 08c46bdf4dcd4350ac16be278a2b17dd (127.8.254.65:39321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3305374c94c4d92aafa1c85a8767958: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
I20260924 01:02:46.883919 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.881676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.8.254.66:38589)
0924 01:02:46.881886 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.881893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.881997 (+ 104us) server_negotiation.cc:407] Connection header received
0924 01:02:46.882218 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.882223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.882333 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.882446 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.883341 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.883346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.883349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.883363 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.883405 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.883416 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.883516 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.883557 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.883600 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 01:02:46.884502 14235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:46.936513 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.936044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46896)
0924 01:02:46.936263 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.936369 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 01:02:46.937589 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.937191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46910)
0924 01:02:46.937345 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.937427 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 01:02:46.972193 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14113
I20260924 01:02:46.989476 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14243
I20260924 01:02:46.992431 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.990384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.8.254.67:35177)
0924 01:02:46.990563 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:46.990568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:46.990585 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:02:46.990643 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:46.990647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:46.990737 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:46.990835 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:46.991758 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:46.991762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:46.991765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:46.991781 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:46.991814 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:46.991825 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:46.991932 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:46.992033 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 01:02:46.992081 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:02:46.993177 14365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:46.998472 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.997989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46912)
0924 01:02:46.998182 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.998333 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 01:02:46.999507 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:46.999127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46922)
0924 01:02:46.999301 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:46.999382 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 01:02:47.080214 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:47.079723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46938)
0924 01:02:47.079926 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:47.080037 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 01:02:47.080703 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14243
I20260924 01:02:47.087142 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:47.086521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39321 (local address 127.0.0.1:46944)
0924 01:02:47.086798 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:47.086912 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 01:02:47.098042 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 13892
I20260924 01:02:47.099386 13953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:47.204843 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 13892
2026-09-24T01:02:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3054 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 01:02:47.246573 9209 mini_oidc.cc:152] Starting JWKS server
I20260924 01:02:47.249207 9209 webserver.cc:460] Webserver started at https://127.0.0.1:40789/ using document root <none> and password file <none>
I20260924 01:02:47.249523 9209 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 01:02:47.250234 9209 webserver.cc:460] Webserver started at http://127.0.0.1:46335/ using document root <none> and password file <none>
2026-09-24T01:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:47Z Disabled control of system clock
I20260924 01:02:47.277884 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38095
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38095
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/testchainca.cert with env {}
W20260924 01:02:47.604516 14431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:47.604802 14431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:47.604889 14431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:47.612069 14431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:47.612174 14431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:47.612211 14431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:47.612259 14431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:47.612304 14431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 01:02:47.612365 14431 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 01:02:47.612427 14431 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40789/jwks.json
I20260924 01:02:47.622243 14431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:47.623524 14431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:47.624786 14431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:47.637259 14436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:47.637180 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:47.637178 14437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:47.638326 14431 server_base.cc:1034] running on GCE node
I20260924 01:02:47.639173 14431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:47.640668 14431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:47.641875 14431 hybrid_clock.cc:648] HybridClock initialized: now 1790211767641809 us; error 70 us; skew 500 ppm
I20260924 01:02:47.642299 14431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:47.644724 14431 webserver.cc:460] Webserver started at http://127.8.254.126:41883/ using document root <none> and password file <none>
I20260924 01:02:47.645455 14431 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:47.645555 14431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:47.645905 14431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:47.649340 14431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba"
format_stamp: "Formatted at 2026-09-24 01:02:47 on dist-test-slave-5c7l"
server_key: "86042a07daf9a683c63ba5a4af85a6d4"
server_key_iv: "a77ea03d574c0758effc541a515215d2"
server_key_version: "encryptionkey@0"
I20260924 01:02:47.650185 14431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba"
format_stamp: "Formatted at 2026-09-24 01:02:47 on dist-test-slave-5c7l"
server_key: "86042a07daf9a683c63ba5a4af85a6d4"
server_key_iv: "a77ea03d574c0758effc541a515215d2"
server_key_version: "encryptionkey@0"
I20260924 01:02:47.655630 14431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 01:02:47.659159 14445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:47.660569 14431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 01:02:47.660768 14431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba"
format_stamp: "Formatted at 2026-09-24 01:02:47 on dist-test-slave-5c7l"
server_key: "86042a07daf9a683c63ba5a4af85a6d4"
server_key_iv: "a77ea03d574c0758effc541a515215d2"
server_key_version: "encryptionkey@0"
I20260924 01:02:47.661003 14431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:47.683049 14431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:47.683918 14431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:47.684245 14431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:47.702584 14431 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:38095
I20260924 01:02:47.702602 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:38095 every 8 connection(s)
I20260924 01:02:47.703926 14431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:47.707057 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14431
I20260924 01:02:47.707257 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:47.707764 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:47.707968 9209 external_mini_cluster.cc:1428] Setting key ac2e002df0d38ca9ec118f8e85af8cfe
I20260924 01:02:47.723405 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:47.709586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.8.254.126:38095)
0924 01:02:47.710616 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:47.710630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:47.710680 (+ 50us) server_negotiation.cc:407] Connection header received
0924 01:02:47.713112 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:47.713142 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:47.713777 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:47.714257 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:47.714879 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:47.716887 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:47.718256 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:47.719280 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:47.719486 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:47.720595 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:47.720609 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:47.720628 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:47.721356 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:47.721462 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:47.721487 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:47.721596 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:47.721797 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 01:02:47.722109 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":292,"threads_started":1}
I20260924 01:02:47.723536 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba: Bootstrap starting.
I20260924 01:02:47.727190 14497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:47.727603 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:02:47.728291 14497 log.cc:824] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:47.731556 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba: No bootstrap required, opened a new log
I20260924 01:02:47.739075 14497 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38095 } }
I20260924 01:02:47.739445 14497 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:47.739569 14497 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff860bfe2a2e4c7f8d9396658a3e39ba, State: Initialized, Role: FOLLOWER
I20260924 01:02:47.740139 14497 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38095 } }
I20260924 01:02:47.740375 14497 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:47.740514 14497 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:47.740687 14497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:47.742954 14497 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38095 } }
I20260924 01:02:47.743453 14497 leader_election.cc:302] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff860bfe2a2e4c7f8d9396658a3e39ba; no voters:
I20260924 01:02:47.744047 14497 leader_election.cc:288] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:47.744161 14502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:47.745414 14502 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [term 1 LEADER]: Becoming Leader. State: Replica: ff860bfe2a2e4c7f8d9396658a3e39ba, State: Running, Role: LEADER
I20260924 01:02:47.745913 14502 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38095 } }
I20260924 01:02:47.746383 14497 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:47.748656 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff860bfe2a2e4c7f8d9396658a3e39ba. Latest consensus state: current_term: 1 leader_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38095 } } }
I20260924 01:02:47.748682 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff860bfe2a2e4c7f8d9396658a3e39ba" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38095 } } }
I20260924 01:02:47.749142 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:47.749225 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:47.761757 14507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:47.766429 14507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:47.774602 14507 catalog_manager.cc:1347] Generated new cluster ID: ed451e21713b46b8b25c49b7eba2d9c0
I20260924 01:02:47.774735 14507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:47.787443 14507 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:47.789320 14507 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:47.803179 14507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff860bfe2a2e4c7f8d9396658a3e39ba: Generated new TSK 0
I20260924 01:02:47.804275 14507 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 01:02:47.852347 14527 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 01:02:47.852581 14527 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 01:02:47.853302 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:47.824220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.8.254.126:38095)
0924 01:02:47.824465 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:47.824472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:47.824892 (+ 420us) server_negotiation.cc:407] Connection header received
0924 01:02:47.825215 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:47.825219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:47.825297 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:47.825418 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0924 01:02:47.826059 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:47.826872 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:47.828982 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:47.829409 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:47.829496 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:47.830138 (+ 642us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 01:02:47.852738 (+ 22600us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 01:02:47.853042 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 01:02:47.853442 14500 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 01:02:47.855337 14526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:47.824118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38095 (local address 127.0.0.1:50036)
0924 01:02:47.824778 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:47.824799 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:47.824995 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:47.825483 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:47.825489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:47.825502 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 01:02:47.825885 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:47.825899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:47.827011 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:47.827016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:47.828828 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:47.828857 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:47.829583 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:47.829590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:47.829769 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:47.829844 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 01:02:47.853048 (+ 23204us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 01:02:47.853444 (+ 396us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.254.126:38095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":316,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.254.126:38095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 01:02:47.857455 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14431
I20260924 01:02:47.858722 14492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:47.952553 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14431
2026-09-24T01:02:47Z chronyd exiting
I20260924 01:02:48.003592 9209 test_util.cc:175] -----------------------------------------------
I20260924 01:02:48.003718 9209 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790211722208042-9209-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (761 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 01:02:48.007572 9209 mini_oidc.cc:152] Starting JWKS server
I20260924 01:02:48.009413 9209 webserver.cc:460] Webserver started at https://127.0.0.1:46849/ using document root <none> and password file <none>
I20260924 01:02:48.009694 9209 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 01:02:48.010427 9209 webserver.cc:460] Webserver started at http://127.0.0.1:41289/ using document root <none> and password file <none>
2026-09-24T01:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:48Z Disabled control of system clock
I20260924 01:02:48.037581 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39353
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39353
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/testchainca.cert with env {}
W20260924 01:02:48.361142 14534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:48.361447 14534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:48.361536 14534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:48.368970 14534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:48.369094 14534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:48.369136 14534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:48.369186 14534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:48.369230 14534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 01:02:48.369285 14534 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 01:02:48.369318 14534 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46849/jwks.json
I20260924 01:02:48.379508 14534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:48.380765 14534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:48.382056 14534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:48.393879 14542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:48.393980 14540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:48.393961 14539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:48.394750 14534 server_base.cc:1034] running on GCE node
I20260924 01:02:48.395619 14534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:48.397154 14534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:48.398432 14534 hybrid_clock.cc:648] HybridClock initialized: now 1790211768398371 us; error 51 us; skew 500 ppm
I20260924 01:02:48.398854 14534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:48.401309 14534 webserver.cc:460] Webserver started at http://127.8.254.126:37021/ using document root <none> and password file <none>
I20260924 01:02:48.402050 14534 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:48.402235 14534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:48.402588 14534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:48.405592 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "41170cd0b0a143b5985dc77e3f060d5d"
format_stamp: "Formatted at 2026-09-24 01:02:48 on dist-test-slave-5c7l"
server_key: "2cebfcfc180f9181b31e2230dd6c463b"
server_key_iv: "b9f5ae1a951c5786a05289a8c761d8a7"
server_key_version: "encryptionkey@0"
I20260924 01:02:48.406481 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "41170cd0b0a143b5985dc77e3f060d5d"
format_stamp: "Formatted at 2026-09-24 01:02:48 on dist-test-slave-5c7l"
server_key: "2cebfcfc180f9181b31e2230dd6c463b"
server_key_iv: "b9f5ae1a951c5786a05289a8c761d8a7"
server_key_version: "encryptionkey@0"
I20260924 01:02:48.411695 14534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 01:02:48.415225 14548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:48.416651 14534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 01:02:48.416859 14534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "41170cd0b0a143b5985dc77e3f060d5d"
format_stamp: "Formatted at 2026-09-24 01:02:48 on dist-test-slave-5c7l"
server_key: "2cebfcfc180f9181b31e2230dd6c463b"
server_key_iv: "b9f5ae1a951c5786a05289a8c761d8a7"
server_key_version: "encryptionkey@0"
I20260924 01:02:48.417084 14534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:48.429813 14534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:48.430680 14534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:48.430987 14534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:48.449123 14534 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39353
I20260924 01:02:48.449131 14599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39353 every 8 connection(s)
I20260924 01:02:48.450588 14534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:48.454385 14600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:48.456409 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14534
I20260924 01:02:48.456630 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:48.457283 9209 external_mini_cluster.cc:1428] Setting key 06c1d6d63225bbab9934081af7466c11
I20260924 01:02:48.466351 14600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d: Bootstrap starting.
I20260924 01:02:48.470048 14600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:48.470913 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:48.459201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.8.254.126:39353)
0924 01:02:48.460217 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:48.460229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:48.460276 (+ 47us) server_negotiation.cc:407] Connection header received
0924 01:02:48.461872 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:48.461895 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:48.462454 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:48.462800 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:48.463377 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:48.464671 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:48.466524 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:48.467177 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:48.467344 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:48.468502 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:48.468512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:48.468525 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:48.469021 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:48.469111 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:48.469134 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:48.469229 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:48.469463 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 01:02:48.469782 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":376,"threads_started":1}
I20260924 01:02:48.471230 14600 log.cc:824] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:48.474565 14600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d: No bootstrap required, opened a new log
I20260924 01:02:48.475231 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:02:48.483108 14600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41170cd0b0a143b5985dc77e3f060d5d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39353 } }
I20260924 01:02:48.483446 14600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:48.483521 14600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41170cd0b0a143b5985dc77e3f060d5d, State: Initialized, Role: FOLLOWER
I20260924 01:02:48.484074 14600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41170cd0b0a143b5985dc77e3f060d5d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39353 } }
I20260924 01:02:48.484278 14600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:48.484377 14600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:48.484504 14600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:48.486740 14600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41170cd0b0a143b5985dc77e3f060d5d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39353 } }
I20260924 01:02:48.487176 14600 leader_election.cc:302] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41170cd0b0a143b5985dc77e3f060d5d; no voters:
I20260924 01:02:48.487686 14600 leader_election.cc:288] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:48.487916 14605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:48.488634 14605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [term 1 LEADER]: Becoming Leader. State: Replica: 41170cd0b0a143b5985dc77e3f060d5d, State: Running, Role: LEADER
I20260924 01:02:48.489167 14605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41170cd0b0a143b5985dc77e3f060d5d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39353 } }
I20260924 01:02:48.489979 14600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:48.492062 14607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41170cd0b0a143b5985dc77e3f060d5d. Latest consensus state: current_term: 1 leader_uuid: "41170cd0b0a143b5985dc77e3f060d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41170cd0b0a143b5985dc77e3f060d5d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39353 } } }
I20260924 01:02:48.492071 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41170cd0b0a143b5985dc77e3f060d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41170cd0b0a143b5985dc77e3f060d5d" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39353 } } }
I20260924 01:02:48.492547 14607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:48.492547 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:48.494808 14610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:48.500133 14610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:48.511040 14610 catalog_manager.cc:1347] Generated new cluster ID: 4dc801e7ea7b49da97313ce3f8da2793
I20260924 01:02:48.511178 14610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:48.520679 14610 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:48.522456 14610 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:48.531334 14610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41170cd0b0a143b5985dc77e3f060d5d: Generated new TSK 0
I20260924 01:02:48.532240 14610 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 01:02:48.583230 14630 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 01:02:48.583371 14630 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 01:02:48.584306 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:48.556477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.8.254.126:39353)
0924 01:02:48.556732 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:48.556740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:48.557104 (+ 364us) server_negotiation.cc:407] Connection header received
0924 01:02:48.557366 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:48.557370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:48.557453 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:48.557582 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0924 01:02:48.558115 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:48.558916 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:48.560921 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:48.561341 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:48.561431 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:48.562004 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 01:02:48.583740 (+ 21736us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 01:02:48.584079 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 01:02:48.584479 14603 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 01:02:48.584721 14629 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:48.556350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39353 (local address 127.0.0.1:54916)
0924 01:02:48.556988 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:48.557013 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:48.557170 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:48.557626 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:48.557631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:48.557645 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0924 01:02:48.557950 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:48.557967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:48.559062 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:48.559068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:48.560776 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:48.560791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:48.561499 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:48.561505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:48.561697 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:48.561742 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 01:02:48.584012 (+ 22270us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 01:02:48.584317 (+ 305us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.254.126:39353: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":342,"threads_started":1}
I20260924 01:02:48.586053 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14534
I20260924 01:02:48.587270 14595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:48.680500 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14534
2026-09-24T01:02:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 01:02:48.736740 9209 mini_oidc.cc:152] Starting JWKS server
I20260924 01:02:48.738651 9209 webserver.cc:460] Webserver started at https://127.0.0.1:46695/ using document root <none> and password file <none>
I20260924 01:02:48.738900 9209 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 01:02:48.739529 9209 webserver.cc:460] Webserver started at http://127.0.0.1:39613/ using document root <none> and password file <none>
2026-09-24T01:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:48Z Disabled control of system clock
I20260924 01:02:48.766757 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45699
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45699
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 01:02:49.102501 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:49.102846 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:49.103005 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:49.110250 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:49.110396 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:49.110504 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:49.110612 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:49.110699 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 01:02:49.110785 14637 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 01:02:49.110841 14637 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46695/jwks.json
I20260924 01:02:49.119879 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:49.121281 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:49.123100 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:49.134917 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:49.134971 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:49.134886 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:49.136348 14637 server_base.cc:1034] running on GCE node
I20260924 01:02:49.137169 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:49.138682 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:49.139917 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790211769139877 us; error 37 us; skew 500 ppm
I20260924 01:02:49.140296 14637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:49.142758 14637 webserver.cc:460] Webserver started at http://127.8.254.126:33031/ using document root <none> and password file <none>
I20260924 01:02:49.143465 14637 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:49.143599 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:49.143935 14637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:49.146942 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "1ea7500d1b61495e9c6469d597349c71"
format_stamp: "Formatted at 2026-09-24 01:02:49 on dist-test-slave-5c7l"
server_key: "a144dd1a3d5a2b68093e1572af3a33c9"
server_key_iv: "b66d2e62cd7ba8b19b13046433c8247f"
server_key_version: "encryptionkey@0"
I20260924 01:02:49.147751 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "1ea7500d1b61495e9c6469d597349c71"
format_stamp: "Formatted at 2026-09-24 01:02:49 on dist-test-slave-5c7l"
server_key: "a144dd1a3d5a2b68093e1572af3a33c9"
server_key_iv: "b66d2e62cd7ba8b19b13046433c8247f"
server_key_version: "encryptionkey@0"
I20260924 01:02:49.153069 14637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:49.156662 14651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:49.157981 14637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:02:49.158180 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "1ea7500d1b61495e9c6469d597349c71"
format_stamp: "Formatted at 2026-09-24 01:02:49 on dist-test-slave-5c7l"
server_key: "a144dd1a3d5a2b68093e1572af3a33c9"
server_key_iv: "b66d2e62cd7ba8b19b13046433c8247f"
server_key_version: "encryptionkey@0"
I20260924 01:02:49.158480 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:49.187878 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:49.188774 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:49.189091 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:49.208223 14637 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45699
I20260924 01:02:49.208248 14702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45699 every 8 connection(s)
I20260924 01:02:49.209596 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:49.213347 14703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:49.216080 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14637
I20260924 01:02:49.216282 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:49.217082 9209 external_mini_cluster.cc:1428] Setting key 8b6ef7301770014223143f58851019e3
I20260924 01:02:49.225117 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71: Bootstrap starting.
I20260924 01:02:49.228462 14703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:49.229596 14703 log.cc:824] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:49.230705 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:49.219127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.8.254.126:45699)
0924 01:02:49.220095 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:49.220107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:49.220145 (+ 38us) server_negotiation.cc:407] Connection header received
0924 01:02:49.221703 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:49.221724 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:49.222255 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:49.222686 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:49.223267 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.224465 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.225915 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.226645 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.226846 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:49.228023 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:49.228031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:49.228046 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:49.228626 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:49.228722 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:49.228747 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:49.228895 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:49.229105 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 01:02:49.229418 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":324,"threads_started":1}
I20260924 01:02:49.232843 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71: No bootstrap required, opened a new log
I20260924 01:02:49.234761 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:02:49.241677 14703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea7500d1b61495e9c6469d597349c71" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45699 } }
I20260924 01:02:49.242058 14703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:49.242172 14703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7500d1b61495e9c6469d597349c71, State: Initialized, Role: FOLLOWER
I20260924 01:02:49.242774 14703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea7500d1b61495e9c6469d597349c71" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45699 } }
I20260924 01:02:49.243007 14703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:49.243156 14703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:49.243327 14703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:49.245569 14703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea7500d1b61495e9c6469d597349c71" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45699 } }
I20260924 01:02:49.246253 14703 leader_election.cc:302] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ea7500d1b61495e9c6469d597349c71; no voters:
I20260924 01:02:49.246986 14703 leader_election.cc:288] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:49.247080 14708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:49.247901 14708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea7500d1b61495e9c6469d597349c71, State: Running, Role: LEADER
I20260924 01:02:49.248450 14708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea7500d1b61495e9c6469d597349c71" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45699 } }
I20260924 01:02:49.249264 14703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:49.251456 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea7500d1b61495e9c6469d597349c71. Latest consensus state: current_term: 1 leader_uuid: "1ea7500d1b61495e9c6469d597349c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea7500d1b61495e9c6469d597349c71" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45699 } } }
I20260924 01:02:49.251808 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:49.252278 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ea7500d1b61495e9c6469d597349c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea7500d1b61495e9c6469d597349c71" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45699 } } }
I20260924 01:02:49.252596 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:49.255198 14714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:49.260805 14714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:49.270002 14714 catalog_manager.cc:1347] Generated new cluster ID: 1c982c3250ac492790567d7235aa5dfb
I20260924 01:02:49.270134 14714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:49.295980 14714 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:49.297808 14714 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:49.307173 14714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ea7500d1b61495e9c6469d597349c71: Generated new TSK 0
I20260924 01:02:49.308291 14714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:02:49.342293 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:49.333587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.8.254.126:45699)
0924 01:02:49.333792 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:49.333798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:49.334352 (+ 554us) server_negotiation.cc:407] Connection header received
0924 01:02:49.334582 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:49.334587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:49.334679 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:49.334860 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0924 01:02:49.335289 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.336419 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.337850 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.338549 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.338675 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:49.339227 (+ 552us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 01:02:49.341588 (+ 2361us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 01:02:49.341916 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 01:02:49.342428 14706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 01:02:49.344158 14732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:49.333505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45699 (local address 127.0.0.1:54218)
0924 01:02:49.334240 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:49.334261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:49.334394 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:49.334881 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:49.334885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:49.334895 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 01:02:49.335133 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:49.335140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.336602 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.336606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:49.337704 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:49.337717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.338759 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.338764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:49.338906 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:49.338939 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 01:02:49.341801 (+ 2862us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 01:02:49.343750 (+ 1949us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.254.126:45699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":1632,"thread_start_us":387,"threads_started":1}
I20260924 01:02:49.345479 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14637
I20260924 01:02:49.346800 14698 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 01:02:49.358299 14733 webserver.cc:533] Webserver: sslize failed:
E20260924 01:02:49.358455 14733 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 01:02:49.442474 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14637
2026-09-24T01:02:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (762 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T01:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:49Z Disabled control of system clock
I20260924 01:02:49.527726 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45043
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45043
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/testchainca.cert with env {}
W20260924 01:02:49.854651 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:49.855002 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:49.855161 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:49.862675 14738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:49.862793 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:49.862887 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:49.863003 14738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:49.863086 14738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 01:02:49.863189 14738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 01:02:49.863263 14738 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 01:02:49.873451 14738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:49.874818 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:49.876120 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:49.888362 14743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:49.888401 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:49.888326 14746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:49.888940 14738 server_base.cc:1034] running on GCE node
I20260924 01:02:49.889719 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:49.891155 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:49.892402 14738 hybrid_clock.cc:648] HybridClock initialized: now 1790211769892353 us; error 61 us; skew 500 ppm
I20260924 01:02:49.892761 14738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:02:49.895207 14738 webserver.cc:460] Webserver started at http://127.8.254.126:39745/ using document root <none> and password file <none>
I20260924 01:02:49.895921 14738 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:49.896061 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:49.896386 14738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:49.899447 14738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "bd1fe4247d6445ffbc761ab7a99bd123"
format_stamp: "Formatted at 2026-09-24 01:02:49 on dist-test-slave-5c7l"
server_key: "968435e4e9ea5b05801236e05d921f20"
server_key_iv: "c3055c4ffc5b482c683132dbecc230f8"
server_key_version: "encryptionkey@0"
I20260924 01:02:49.900266 14738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "bd1fe4247d6445ffbc761ab7a99bd123"
format_stamp: "Formatted at 2026-09-24 01:02:49 on dist-test-slave-5c7l"
server_key: "968435e4e9ea5b05801236e05d921f20"
server_key_iv: "c3055c4ffc5b482c683132dbecc230f8"
server_key_version: "encryptionkey@0"
I20260924 01:02:49.905457 14738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:49.908957 14752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:49.910395 14738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:49.910593 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "bd1fe4247d6445ffbc761ab7a99bd123"
format_stamp: "Formatted at 2026-09-24 01:02:49 on dist-test-slave-5c7l"
server_key: "968435e4e9ea5b05801236e05d921f20"
server_key_iv: "c3055c4ffc5b482c683132dbecc230f8"
server_key_version: "encryptionkey@0"
I20260924 01:02:49.910845 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:49.935292 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:49.936213 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:49.936518 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:49.955309 14738 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45043
I20260924 01:02:49.955314 14803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45043 every 8 connection(s)
I20260924 01:02:49.956833 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:49.959470 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14738
I20260924 01:02:49.959694 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:49.960522 9209 external_mini_cluster.cc:1428] Setting key bcae1fcec3c0712faa381cca77b8350a
I20260924 01:02:49.961381 14804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:49.975116 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:49.962559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.8.254.126:45043)
0924 01:02:49.963586 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:49.963597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:49.963638 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:02:49.965865 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:49.965887 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:49.966480 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:49.966828 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:49.967444 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.968796 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.970616 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:49.971258 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:49.971416 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:49.972666 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:49.972676 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:49.972690 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:02:49.973301 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:49.973384 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:02:49.973405 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:49.973496 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:49.973729 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 01:02:49.974083 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":400,"threads_started":1}
I20260924 01:02:49.975433 14804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123: Bootstrap starting.
I20260924 01:02:49.979276 14804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:49.979418 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:02:49.980536 14804 log.cc:824] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:49.983695 14804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123: No bootstrap required, opened a new log
I20260924 01:02:49.991050 14804 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45043 } }
I20260924 01:02:49.991411 14804 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:49.991488 14804 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1fe4247d6445ffbc761ab7a99bd123, State: Initialized, Role: FOLLOWER
I20260924 01:02:49.992062 14804 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45043 } }
I20260924 01:02:49.992269 14804 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:49.992370 14804 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:49.992494 14804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:49.994807 14804 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45043 } }
I20260924 01:02:49.995285 14804 leader_election.cc:302] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1fe4247d6445ffbc761ab7a99bd123; no voters:
I20260924 01:02:49.995872 14804 leader_election.cc:288] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:49.996055 14809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:49.996840 14809 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [term 1 LEADER]: Becoming Leader. State: Replica: bd1fe4247d6445ffbc761ab7a99bd123, State: Running, Role: LEADER
I20260924 01:02:49.997481 14809 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45043 } }
I20260924 01:02:49.998356 14804 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:50.000835 14811 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1fe4247d6445ffbc761ab7a99bd123. Latest consensus state: current_term: 1 leader_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45043 } } }
I20260924 01:02:50.001358 14811 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:50.000936 14810 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1fe4247d6445ffbc761ab7a99bd123" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45043 } } }
I20260924 01:02:50.001595 14810 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:50.002942 14818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:50.008471 14818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:50.018110 14818 catalog_manager.cc:1347] Generated new cluster ID: d8653a2f1c2c47d9a89a523e2ced1a91
I20260924 01:02:50.018265 14818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:50.042429 14818 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:50.044409 14818 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:50.058533 14818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd1fe4247d6445ffbc761ab7a99bd123: Generated new TSK 0
I20260924 01:02:50.059551 14818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:02:50.602892 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:50.072979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.8.254.126:45043)
0924 01:02:50.073200 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:50.073208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:50.073655 (+ 447us) server_negotiation.cc:407] Connection header received
0924 01:02:50.073859 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:50.073867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:50.073974 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:50.074149 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0924 01:02:50.074649 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:50.075792 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:50.077148 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:50.077855 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:50.077979 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:50.078480 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 01:02:50.602246 (+523766us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 01:02:50.602610 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55422: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 01:02:50.602998 14807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55422: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 01:02:50.603359 14833 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:50.072838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45043 (local address 127.0.0.1:55422)
0924 01:02:50.073554 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:50.073571 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:50.073693 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:50.074154 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:50.074157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:50.074166 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 01:02:50.074501 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:50.074509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:50.075933 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:50.075937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:50.077009 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:50.077021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:50.078017 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:50.078021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:50.078137 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:50.078169 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 01:02:50.602549 (+524380us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 01:02:50.602878 (+ 329us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.254.126:45043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":416,"threads_started":1}
I20260924 01:02:50.604758 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14738
I20260924 01:02:50.605866 14799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:50.710182 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14738
2026-09-24T01:02:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1248 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (415 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (414 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:51 dist-test-slave-5c7l krb5kdc[14848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:51 dist-test-slave-5c7l krb5kdc[14848](info): set up 2 sockets
Sep 24 01:02:51 dist-test-slave-5c7l krb5kdc[14848](info): commencing operation
krb5kdc: starting...
W20260924 01:02:52.652031 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:52 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:02:52.866683 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40531
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:40531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:53.194979 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:53.195295 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:53.195474 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:53.203503 14861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:53.203617 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:53.203754 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:53.203857 14861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:53.203953 14861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:53.214917 14861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:40531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:53.216354 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:53.217758 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:53.230352 14869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:53.230444 14867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:53.230422 14866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:53.231750 14861 server_base.cc:1034] running on GCE node
I20260924 01:02:53.232667 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:53.234172 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:53.235491 14861 hybrid_clock.cc:648] HybridClock initialized: now 1790211773235434 us; error 58 us; skew 500 ppm
I20260924 01:02:53.235857 14861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:53 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:53.239092 14861 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:02:53.240634 14861 webserver.cc:460] Webserver started at http://127.8.254.126:38061/ using document root <none> and password file <none>
I20260924 01:02:53.241293 14861 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:53.241403 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:53.241688 14861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:53.244720 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "c823952944dd4561b16d07b967a16fdf"
format_stamp: "Formatted at 2026-09-24 01:02:53 on dist-test-slave-5c7l"
server_key: "fd70a1c7d7a084a88d74e69901d64b79"
server_key_iv: "0800afb434a57e9ce9b61354597956f3"
server_key_version: "encryptionkey@0"
I20260924 01:02:53.245464 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "c823952944dd4561b16d07b967a16fdf"
format_stamp: "Formatted at 2026-09-24 01:02:53 on dist-test-slave-5c7l"
server_key: "fd70a1c7d7a084a88d74e69901d64b79"
server_key_iv: "0800afb434a57e9ce9b61354597956f3"
server_key_version: "encryptionkey@0"
I20260924 01:02:53.250679 14861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 01:02:53.254066 14876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:53.255435 14861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:53.255582 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "c823952944dd4561b16d07b967a16fdf"
format_stamp: "Formatted at 2026-09-24 01:02:53 on dist-test-slave-5c7l"
server_key: "fd70a1c7d7a084a88d74e69901d64b79"
server_key_iv: "0800afb434a57e9ce9b61354597956f3"
server_key_version: "encryptionkey@0"
I20260924 01:02:53.255751 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:53.304557 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:53.309440 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:53.309686 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:53.328528 14861 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:40531
I20260924 01:02:53.328567 14927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:40531 every 8 connection(s)
I20260924 01:02:53.329978 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:53.333827 14928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:53.338421 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14861
I20260924 01:02:53.338672 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:53.339443 9209 external_mini_cluster.cc:1428] Setting key d75a8bedfd8aae82a75eccb32bfc6153
I20260924 01:02:53.345808 14928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf: Bootstrap starting.
I20260924 01:02:53.349121 14928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:53.350241 14928 log.cc:824] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:53.353173 14928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf: No bootstrap required, opened a new log
Sep 24 01:02:53 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:53.362015 14928 raft_consensus.cc:348] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c823952944dd4561b16d07b967a16fdf" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40531 } }
I20260924 01:02:53.362538 14928 raft_consensus.cc:374] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:53.362691 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:53.341682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.8.254.126:40531)
0924 01:02:53.342582 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:53.342594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:53.342635 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:02:53.344461 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:53.344485 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:53.344895 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:53.345313 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:53.346907 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:53.348245 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:53.349600 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:53.350440 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:53.350639 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:53.354733 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:53.354742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:53.354760 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:53.354801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:53.358014 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:53.358858 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:53.358879 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:53.358887 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:53.359079 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:53.359545 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:53.359550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:53.359553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:53.360223 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:53.360582 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:53.361007 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 01:02:53.361459 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":413,"threads_started":1}
I20260924 01:02:53.362671 14928 raft_consensus.cc:729] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c823952944dd4561b16d07b967a16fdf, State: Initialized, Role: FOLLOWER
I20260924 01:02:53.363485 14928 consensus_queue.cc:260] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c823952944dd4561b16d07b967a16fdf" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40531 } }
I20260924 01:02:53.363767 14928 raft_consensus.cc:388] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:53.363926 14928 raft_consensus.cc:482] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:53.364121 14928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:53.366734 14928 raft_consensus.cc:504] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c823952944dd4561b16d07b967a16fdf" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40531 } }
I20260924 01:02:53.367272 14928 leader_election.cc:302] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c823952944dd4561b16d07b967a16fdf; no voters:
I20260924 01:02:53.367883 14928 leader_election.cc:288] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:53.368054 14933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:53.368805 14933 raft_consensus.cc:686] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [term 1 LEADER]: Becoming Leader. State: Replica: c823952944dd4561b16d07b967a16fdf, State: Running, Role: LEADER
I20260924 01:02:53.369371 14933 consensus_queue.cc:237] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c823952944dd4561b16d07b967a16fdf" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40531 } }
I20260924 01:02:53.370290 14928 sys_catalog.cc:564] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:53.372637 14934 sys_catalog.cc:455] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c823952944dd4561b16d07b967a16fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c823952944dd4561b16d07b967a16fdf" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40531 } } }
I20260924 01:02:53.373127 14934 sys_catalog.cc:458] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:53.373296 14935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c823952944dd4561b16d07b967a16fdf. Latest consensus state: current_term: 1 leader_uuid: "c823952944dd4561b16d07b967a16fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c823952944dd4561b16d07b967a16fdf" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40531 } } }
I20260924 01:02:53.373714 14935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:53.375730 14941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:53.380952 14941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:53.389649 14941 catalog_manager.cc:1347] Generated new cluster ID: b8364023d0b944adbd43af1a0127ee43
I20260924 01:02:53.389806 14941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:53.405280 14941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:53.406859 14941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:53.421700 14941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c823952944dd4561b16d07b967a16fdf: Generated new TSK 0
I20260924 01:02:53.422842 14941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:02:53.477391 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40531
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:53.816190 14954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:53.816473 14954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:53.816638 14954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:53.824051 14954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:53.824170 14954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:53.824368 14954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:53.834494 14954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:53.835882 14954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:53.837229 14954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:53.850735 14960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:53.850734 14962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:53.851059 14959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:53.851517 14954 server_base.cc:1034] running on GCE node
I20260924 01:02:53.852128 14954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:53.853070 14954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:53.854305 14954 hybrid_clock.cc:648] HybridClock initialized: now 1790211773854252 us; error 37 us; skew 500 ppm
I20260924 01:02:53.854636 14954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:53 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:53.857905 14954 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:02:53.859495 14954 webserver.cc:460] Webserver started at http://127.8.254.65:41367/ using document root <none> and password file <none>
I20260924 01:02:53.860280 14954 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:53.860419 14954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:53.860752 14954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:53.863775 14954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "ea107136337845ada5c78baa05a54291"
format_stamp: "Formatted at 2026-09-24 01:02:53 on dist-test-slave-5c7l"
server_key: "facf7f3dc9d47c170a897b400a94ce3f"
server_key_iv: "276a6adcce7dcb7e990a11da5d23dbd4"
server_key_version: "encryptionkey@0"
I20260924 01:02:53.864560 14954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "ea107136337845ada5c78baa05a54291"
format_stamp: "Formatted at 2026-09-24 01:02:53 on dist-test-slave-5c7l"
server_key: "facf7f3dc9d47c170a897b400a94ce3f"
server_key_iv: "276a6adcce7dcb7e990a11da5d23dbd4"
server_key_version: "encryptionkey@0"
I20260924 01:02:53.869762 14954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:53.873375 14969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:53.874914 14954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:53.875070 14954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "ea107136337845ada5c78baa05a54291"
format_stamp: "Formatted at 2026-09-24 01:02:53 on dist-test-slave-5c7l"
server_key: "facf7f3dc9d47c170a897b400a94ce3f"
server_key_iv: "276a6adcce7dcb7e990a11da5d23dbd4"
server_key_version: "encryptionkey@0"
I20260924 01:02:53.875294 14954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:53.904714 14954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:53.909155 14954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:53.909500 14954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:53.910578 14954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:53.912034 14954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:53.912142 14954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:53.912309 14954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:53.912387 14954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:53.945887 14954 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:46203
I20260924 01:02:53.945981 15081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:46203 every 8 connection(s)
I20260924 01:02:53.947232 14954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:53.957262 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 14954
I20260924 01:02:53.957526 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:53.958282 9209 external_mini_cluster.cc:1428] Setting key d0e55517e3fe563d20a3516a20bee415
Sep 24 01:02:53 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211773, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:53.968966 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:53.950023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:36203 (local address 127.8.254.126:40531)
0924 01:02:53.950513 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:53.950517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:53.951585 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 01:02:53.953506 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:53.953511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:53.953614 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:53.953735 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:53.955439 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:53.956178 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:53.957938 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:53.958483 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:53.958622 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:53.962915 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:53.962924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:53.962934 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:53.962973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:53.965603 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:53.966484 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:53.966495 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:53.966497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:53.966620 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:53.967095 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:53.967099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:53.967101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:53.967392 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:53.967553 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:53.968351 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 01:02:53.968600 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":291,"threads_started":1}
I20260924 01:02:53.970028 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:53.950670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40531 (local address 127.8.254.65:36203)
0924 01:02:53.951421 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:53.951460 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:53.953199 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:53.954005 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:53.954014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:53.954626 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:53.955244 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:53.955262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:53.956344 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:53.956354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:53.957740 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:53.957754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:53.958731 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:53.958740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:53.959059 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:53.959892 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:53.959918 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:53.962662 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:53.965772 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:53.965782 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:53.965801 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:53.966346 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:53.966738 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:53.966743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:53.966746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:53.966961 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:53.967527 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:53.967535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:53.967929 (+ 394us) client_negotiation.cc:769] Sending connection context
0924 01:02:53.968215 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 01:02:53.968796 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
I20260924 01:02:53.971841 15082 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40531
I20260924 01:02:53.972182 15082 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:53.972950 15082 heartbeater.cc:511] Master 127.8.254.126:40531 requested a full tablet report, sending...
I20260924 01:02:53.975667 14893 ts_manager.cc:194] Registered new tserver with Master: ea107136337845ada5c78baa05a54291 (127.8.254.65:46203)
I20260924 01:02:53.978009 14893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:36203
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:02:54.006599 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40531
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:54.341840 15088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:54.342183 15088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:54.342387 15088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:54.349933 15088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:54.350100 15088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:54.350332 15088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:54.360834 15088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:54.362228 15088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:54.363615 15088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:54.378060 15093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:54.378012 15096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:54.378057 15094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:54.378870 15088 server_base.cc:1034] running on GCE node
I20260924 01:02:54.379557 15088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:54.380594 15088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:54.381832 15088 hybrid_clock.cc:648] HybridClock initialized: now 1790211774381762 us; error 68 us; skew 500 ppm
I20260924 01:02:54.382179 15088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:54 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:54.385392 15088 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:02:54.387055 15088 webserver.cc:460] Webserver started at http://127.8.254.66:42343/ using document root <none> and password file <none>
I20260924 01:02:54.387763 15088 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:54.387914 15088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:54.388226 15088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:54.391278 15088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "5d090b5487544e6c983d97069896f0d6"
format_stamp: "Formatted at 2026-09-24 01:02:54 on dist-test-slave-5c7l"
server_key: "25607bbdfd0192e2e0dbad63cd4815e0"
server_key_iv: "0df23db1b0acaaddb42b4edb4ad6fd8a"
server_key_version: "encryptionkey@0"
I20260924 01:02:54.392081 15088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "5d090b5487544e6c983d97069896f0d6"
format_stamp: "Formatted at 2026-09-24 01:02:54 on dist-test-slave-5c7l"
server_key: "25607bbdfd0192e2e0dbad63cd4815e0"
server_key_iv: "0df23db1b0acaaddb42b4edb4ad6fd8a"
server_key_version: "encryptionkey@0"
I20260924 01:02:54.397460 15088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:54.401135 15103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:54.402772 15088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:54.402981 15088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "5d090b5487544e6c983d97069896f0d6"
format_stamp: "Formatted at 2026-09-24 01:02:54 on dist-test-slave-5c7l"
server_key: "25607bbdfd0192e2e0dbad63cd4815e0"
server_key_iv: "0df23db1b0acaaddb42b4edb4ad6fd8a"
server_key_version: "encryptionkey@0"
I20260924 01:02:54.403208 15088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:54.423036 15088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:54.427322 15088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:54.427639 15088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:54.428629 15088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:54.430078 15088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:54.430188 15088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:54.430361 15088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:54.430442 15088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:54.464459 15088 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:36111
I20260924 01:02:54.464504 15215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:36111 every 8 connection(s)
I20260924 01:02:54.465811 15088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:54.467749 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15088
I20260924 01:02:54.467989 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:54.468559 9209 external_mini_cluster.cc:1428] Setting key 0f4a5197d72bb8c8caf18749e7623fca
Sep 24 01:02:54 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:54.487833 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:54.468731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:49067 (local address 127.8.254.126:40531)
0924 01:02:54.468918 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:54.468923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:54.470411 (+ 1488us) server_negotiation.cc:407] Connection header received
0924 01:02:54.472396 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:54.472406 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:54.472510 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:54.472682 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:54.474271 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.475627 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.477473 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.477917 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.478115 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:54.481936 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:54.481942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:54.481945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:54.481978 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:54.484701 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:54.485454 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:54.485462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:54.485466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:54.485605 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:54.486055 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:54.486061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:54.486064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:54.486393 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:54.486553 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:54.487251 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 01:02:54.487483 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:02:54.488821 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:54.469349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40531 (local address 127.8.254.66:49067)
0924 01:02:54.470241 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:54.470295 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:54.472080 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:54.472880 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:54.472889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:54.473482 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:54.474053 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:54.474071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.475802 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.475808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:54.477307 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:54.477324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.478256 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.478267 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:54.478463 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:54.479229 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:54.479256 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:54.481691 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:54.484856 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:54.484866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:54.484884 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:54.485317 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:54.485724 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:54.485729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:54.485733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:54.485938 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:54.486523 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:54.486531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:54.486892 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 01:02:54.487172 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 01:02:54.487729 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:02:54.490309 15216 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40531
I20260924 01:02:54.490649 15216 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:54.491387 15216 heartbeater.cc:511] Master 127.8.254.126:40531 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:02:54.492983 14893 ts_manager.cc:194] Registered new tserver with Master: 5d090b5487544e6c983d97069896f0d6 (127.8.254.66:36111)
I20260924 01:02:54.493813 14893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:49067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:02:54.517763 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40531
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:54.848361 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:54.848701 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:54.848873 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:54.856263 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:54.856426 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:54.856601 15221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:54.866765 15221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:54.868104 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:54.869446 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:54.882897 15226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:54.882849 15229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:54.882869 15227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:54.883734 15221 server_base.cc:1034] running on GCE node
I20260924 01:02:54.884397 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:54.885444 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:54.886727 15221 hybrid_clock.cc:648] HybridClock initialized: now 1790211774886641 us; error 67 us; skew 500 ppm
I20260924 01:02:54.887121 15221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:54 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:54.890640 15221 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:02:54.892177 15221 webserver.cc:460] Webserver started at http://127.8.254.67:41927/ using document root <none> and password file <none>
I20260924 01:02:54.892927 15221 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:54.893090 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:54.893441 15221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:54.896795 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "a709e8251f1346099296b58a3ed45eaa"
format_stamp: "Formatted at 2026-09-24 01:02:54 on dist-test-slave-5c7l"
server_key: "caeb3f2bd1a291abf78568907a264dfa"
server_key_iv: "210ec03195db0b8004dfa16691b41fcd"
server_key_version: "encryptionkey@0"
I20260924 01:02:54.897667 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "a709e8251f1346099296b58a3ed45eaa"
format_stamp: "Formatted at 2026-09-24 01:02:54 on dist-test-slave-5c7l"
server_key: "caeb3f2bd1a291abf78568907a264dfa"
server_key_iv: "210ec03195db0b8004dfa16691b41fcd"
server_key_version: "encryptionkey@0"
I20260924 01:02:54.903211 15221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:02:54.907018 15236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:54.908525 15221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:54.908715 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "a709e8251f1346099296b58a3ed45eaa"
format_stamp: "Formatted at 2026-09-24 01:02:54 on dist-test-slave-5c7l"
server_key: "caeb3f2bd1a291abf78568907a264dfa"
server_key_iv: "210ec03195db0b8004dfa16691b41fcd"
server_key_version: "encryptionkey@0"
I20260924 01:02:54.908943 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:54.929279 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:54.933920 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:54.934314 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:54.935330 15221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:54.936739 15221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:54.936847 15221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:54.937023 15221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:54.937098 15221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:54.972304 15221 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:45293
I20260924 01:02:54.972457 15348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:45293 every 8 connection(s)
I20260924 01:02:54.973683 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:54.977778 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15221
I20260924 01:02:54.977986 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:54.978515 9209 external_mini_cluster.cc:1428] Setting key e0c11501fb88bb81ddaf42ba500c67d0
I20260924 01:02:54.981711 15082 heartbeater.cc:503] Master 127.8.254.126:40531 was elected leader, sending a full tablet report...
Sep 24 01:02:54 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211774, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:02:54.994807 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:54.976446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:52687 (local address 127.8.254.126:40531)
0924 01:02:54.976626 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:54.976631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:54.977775 (+ 1144us) server_negotiation.cc:407] Connection header received
0924 01:02:54.980219 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:54.980224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:54.980315 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:54.980427 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:54.982482 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.983355 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.984824 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.985262 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.985354 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:54.988640 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:54.988645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:54.988648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:54.988679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:54.991343 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:54.992072 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:54.992077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:54.992081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:54.992245 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:54.992759 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:54.992764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:54.992768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:54.993198 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:54.993351 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:54.994302 (+ 951us) server_negotiation.cc:299] Negotiation successful
0924 01:02:54.994557 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:02:54.995857 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:54.976888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40531 (local address 127.8.254.67:52687)
0924 01:02:54.977626 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:54.977662 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:54.979925 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:54.980737 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:54.980751 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:54.981488 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:54.982298 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:54.982319 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.983485 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.983489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:54.984700 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:54.984712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:54.985447 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:54.985451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:54.985636 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:54.986307 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:54.986341 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:54.988433 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:54.991480 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:54.991487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:54.991503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:54.991932 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:54.992382 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:54.992387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:54.992389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:54.992639 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:54.993358 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:54.993364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:54.993825 (+ 461us) client_negotiation.cc:769] Sending connection context
0924 01:02:54.994169 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 01:02:54.994675 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":277,"threads_started":1}
I20260924 01:02:54.997301 15349 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40531
I20260924 01:02:54.997627 15349 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:54.998488 15349 heartbeater.cc:511] Master 127.8.254.126:40531 requested a full tablet report, sending...
I20260924 01:02:54.999974 14893 ts_manager.cc:194] Registered new tserver with Master: a709e8251f1346099296b58a3ed45eaa (127.8.254.67:45293)
I20260924 01:02:55.000823 14893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:52687
I20260924 01:02:55.007192 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/test-admin.keytab.
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.035975 9209 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 01:02:55.041453 15356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:55.038845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.8.254.65:46203)
0924 01:02:55.039533 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.039540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.039568 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:02:55.039630 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.039636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.039861 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.039994 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.040968 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51344: BlockingRecv error: recv got EOF from 127.0.0.1:51344 (error 108)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
W20260924 01:02:55.041599 9209 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc' of type FILE
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.043275 9209 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.044662 9209 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:55.061304 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:55.046350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.8.254.65:46203)
0924 01:02:55.046511 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.046515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.046899 (+ 384us) server_negotiation.cc:407] Connection header received
0924 01:02:55.047069 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.047075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.047173 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.047262 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.048310 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.049265 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.050555 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.051170 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.051278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:55.055316 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:55.055336 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:55.055357 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:55.055396 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:55.058387 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.059096 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.059105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.059107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.059229 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.059572 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.059576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.059578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.059965 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:55.060433 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:55.060715 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 01:02:55.061025 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 01:02:55.067129 15356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:55.065522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.8.254.65:46203)
0924 01:02:55.065738 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.065743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.065821 (+ 78us) server_negotiation.cc:407] Connection header received
0924 01:02:55.065953 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.065956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.066015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.066093 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.066890 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51352: BlockingRecv error: recv got EOF from 127.0.0.1:51352 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 01:02:55.067591 9209 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc)
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.069094 9209 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.070441 9209 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:55.084694 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:55.071732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.8.254.65:46203)
0924 01:02:55.071945 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.071952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.072207 (+ 255us) server_negotiation.cc:407] Connection header received
0924 01:02:55.072324 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.072327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.072385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.072474 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.073339 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.074025 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.075370 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.075807 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.075945 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:55.079685 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:55.079690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:55.079693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:55.079726 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:55.082119 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.082774 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.082782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.082787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.082920 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.083326 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.083331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.083336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.083616 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:55.083879 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:55.084210 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 01:02:55.084411 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 01:02:55.088819 15356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:02:55.087385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.8.254.65:46203)
0924 01:02:55.087580 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.087585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.087686 (+ 101us) server_negotiation.cc:407] Connection header received
0924 01:02:55.087802 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.087806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.087871 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.087949 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.088601 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51378: BlockingRecv error: recv got EOF from 127.0.0.1:51378 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 01:02:55.088960 9209 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790211722208042-9209-0/krb5kdc/krb5cc)
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.090576 9209 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:55.091835 9209 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:02:55.105285 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:55.093054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.8.254.65:46203)
0924 01:02:55.093264 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.093270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.093431 (+ 161us) server_negotiation.cc:407] Connection header received
0924 01:02:55.093543 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.093547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.093599 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.093674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.094575 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.095262 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.096392 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.096776 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.096866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:55.100172 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:55.100176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:55.100179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:55.100210 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:55.102826 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.103493 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.103496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.103498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.103640 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.104009 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.104012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.104014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.104264 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:55.104494 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:55.104747 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 01:02:55.104943 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:02:55.106894 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14954
I20260924 01:02:55.118757 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:55.107699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.8.254.65:46203)
0924 01:02:55.107830 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.107834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.108189 (+ 355us) server_negotiation.cc:407] Connection header received
0924 01:02:55.108346 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.108349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.108404 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.108483 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.109456 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.110124 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.111318 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.111765 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.111856 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:55.113897 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:55.113901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:55.113905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:55.113977 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:55.116608 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.117173 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.117177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.117179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.117308 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.117607 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.117610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.117612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.117846 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:55.118043 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:55.118328 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 01:02:55.118490 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 01:02:55.119383 15077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:55.206801 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14954
I20260924 01:02:55.225009 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15088
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:02:55.241011 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:55.225984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.8.254.66:36111)
0924 01:02:55.226546 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.226554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.226581 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:02:55.226643 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.226649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.226877 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.227003 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.228036 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.229083 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.230470 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.231043 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.231143 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:55.235151 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:55.235165 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:55.235183 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:55.235224 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:55.238058 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.238846 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.238853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.238856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.238983 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.239365 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.239369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.239371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.239724 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:55.240074 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:55.240426 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 01:02:55.240693 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
I20260924 01:02:55.242388 15211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:55.325608 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15088
I20260924 01:02:55.341987 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15221
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[14848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:02:55.358043 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:55.343011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.8.254.67:45293)
0924 01:02:55.343696 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:55.343704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:55.343733 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:02:55.343792 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:55.343798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:55.344006 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:55.344141 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:55.345270 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.346464 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.347753 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:55.348311 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:55.348408 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:55.352280 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:55.352292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:55.352307 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:55.352349 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:55.355503 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.356140 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.356147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.356149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.356312 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:55.356644 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:55.356647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:55.356649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:55.356980 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:55.357202 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:55.357448 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 01:02:55.357735 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":370,"threads_started":1}
I20260924 01:02:55.359596 15344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:55.443181 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15221
I20260924 01:02:55.459597 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 14861
I20260924 01:02:55.460810 14923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:02:55.570916 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 14861
2026-09-24T01:02:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4034 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[15376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[15376](info): set up 2 sockets
Sep 24 01:02:55 dist-test-slave-5c7l krb5kdc[15376](info): commencing operation
krb5kdc: starting...
W20260924 01:02:56.668311 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:02:56 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:02:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab.
Entry for principal oryx/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab.
I20260924 01:02:56.881613 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38457
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab
--principal=oryx/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:57.221334 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:57.221649 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:57.221839 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:57.230000 15389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:57.230127 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:57.230268 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:57.230393 15389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:57.230481 15389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:57.241333 15389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab
--principal=oryx/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:57.242794 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:57.244184 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:57.256454 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:57.256500 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:57.256515 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:57.257948 15389 server_base.cc:1034] running on GCE node
I20260924 01:02:57.258836 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:57.260318 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:57.261539 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790211777261491 us; error 29 us; skew 500 ppm
I20260924 01:02:57.261924 15389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:57 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211777, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:57.273334 15389 init.cc:377] Logged in from keytab as oryx/127.8.254.126@KRBTEST.COM (short username oryx)
I20260924 01:02:57.274972 15389 webserver.cc:460] Webserver started at http://127.8.254.126:37687/ using document root <none> and password file <none>
I20260924 01:02:57.275653 15389 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:57.275763 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:57.276016 15389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:57.279070 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "b75f7de8941e1e8bd27544f6ef85a0b8"
server_key_iv: "125156fdf805a5e3b5dc0ddb30e43961"
server_key_version: "encryptionkey@0"
I20260924 01:02:57.279806 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "b75f7de8941e1e8bd27544f6ef85a0b8"
server_key_iv: "125156fdf805a5e3b5dc0ddb30e43961"
server_key_version: "encryptionkey@0"
I20260924 01:02:57.285104 15389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:57.288839 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:57.290349 15389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:57.290524 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "b75f7de8941e1e8bd27544f6ef85a0b8"
server_key_iv: "125156fdf805a5e3b5dc0ddb30e43961"
server_key_version: "encryptionkey@0"
I20260924 01:02:57.290732 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:57.305310 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:57.309401 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:57.309701 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:57.328565 15389 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:38457
I20260924 01:02:57.328609 15455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:38457 every 8 connection(s)
I20260924 01:02:57.329993 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:02:57.332696 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15389
I20260924 01:02:57.332947 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:02:57.333599 9209 external_mini_cluster.cc:1428] Setting key 9d7557c2be3434a1f85f6edcc5af8a92
I20260924 01:02:57.334002 15456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:57.349196 15456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Bootstrap starting.
Sep 24 01:02:57 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:02:57.353634 15456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:57.355068 15456 log.cc:824] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:57.358716 15456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: No bootstrap required, opened a new log
I20260924 01:02:57.359791 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:57.335820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.8.254.126:38457)
0924 01:02:57.336654 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:57.336670 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:57.336719 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:02:57.339174 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:57.339206 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:57.339779 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:57.340256 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:57.341650 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:57.343691 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:57.344983 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:57.346145 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:57.346368 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:57.350530 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:57.350543 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:57.350563 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:57.350618 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:57.354703 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:57.355552 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:57.355563 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:57.355573 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:57.355838 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:57.356289 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:57.356297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:57.356300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:57.357086 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:57.357385 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:57.357859 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 01:02:57.358343 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260924 01:02:57.367271 15456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } }
I20260924 01:02:57.367758 15456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:57.367841 15456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd51e6cfd7d4588a8ebdfe215842ac7, State: Initialized, Role: FOLLOWER
I20260924 01:02:57.368468 15456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } }
I20260924 01:02:57.368685 15456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:02:57.368778 15456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:02:57.368927 15456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:57.371261 15456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } }
I20260924 01:02:57.371825 15456 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd51e6cfd7d4588a8ebdfe215842ac7; no voters:
I20260924 01:02:57.372453 15456 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:02:57.372664 15461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:02:57.374379 15461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd51e6cfd7d4588a8ebdfe215842ac7, State: Running, Role: LEADER
I20260924 01:02:57.374924 15461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } }
I20260924 01:02:57.375370 15456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:02:57.377884 15462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } }
I20260924 01:02:57.377928 15463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51e6cfd7d4588a8ebdfe215842ac7. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } }
I20260924 01:02:57.378288 15462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:57.378361 15463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:57.379629 15468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:02:57.386287 15468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:02:57.395998 15468 catalog_manager.cc:1347] Generated new cluster ID: aafcc22a693344fc831348d8a2da7bdb
I20260924 01:02:57.396137 15468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:02:57.407358 15468 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:02:57.408689 15468 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:02:57.421497 15468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Generated new TSK 0
I20260924 01:02:57.422423 15468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab.
Entry for principal oryx/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab.
I20260924 01:02:57.485193 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38457
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:57.818536 15482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:57.818821 15482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:57.818917 15482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:57.826545 15482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:57.826664 15482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:57.826779 15482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:02:57.836396 15482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:57.837662 15482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:57.839000 15482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:57.852425 15487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:57.852478 15488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:57.852356 15490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:57.858516 15482 server_base.cc:1034] running on GCE node
I20260924 01:02:57.859243 15482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:57.860364 15482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:57.861606 15482 hybrid_clock.cc:648] HybridClock initialized: now 1790211777861556 us; error 40 us; skew 500 ppm
I20260924 01:02:57.861944 15482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:57 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211777, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:57.865339 15482 init.cc:377] Logged in from keytab as oryx/127.8.254.65@KRBTEST.COM (short username oryx)
I20260924 01:02:57.867312 15482 webserver.cc:460] Webserver started at http://127.8.254.65:34443/ using document root <none> and password file <none>
I20260924 01:02:57.868103 15482 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:57.868256 15482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:57.868577 15482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:57.871847 15482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "376712ccc3674ae3be2916b277ef8a19"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "a727229019eda9ec072702dd4a93c1d0"
server_key_iv: "593aa4a71fe9f1d9710331d5f6f1dbd3"
server_key_version: "encryptionkey@0"
I20260924 01:02:57.872640 15482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "376712ccc3674ae3be2916b277ef8a19"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "a727229019eda9ec072702dd4a93c1d0"
server_key_iv: "593aa4a71fe9f1d9710331d5f6f1dbd3"
server_key_version: "encryptionkey@0"
I20260924 01:02:57.878391 15482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:02:57.882102 15497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:57.883574 15482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:57.883765 15482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "376712ccc3674ae3be2916b277ef8a19"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "a727229019eda9ec072702dd4a93c1d0"
server_key_iv: "593aa4a71fe9f1d9710331d5f6f1dbd3"
server_key_version: "encryptionkey@0"
I20260924 01:02:57.883983 15482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:57.907593 15482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:57.911988 15482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:57.912294 15482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:57.913416 15482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:57.915021 15482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:57.915131 15482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:57.915310 15482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:57.915400 15482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:57.948812 15482 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:40283
I20260924 01:02:57.948907 15609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:40283 every 8 connection(s)
I20260924 01:02:57.950212 15482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:02:57.960592 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15482
I20260924 01:02:57.960851 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:02:57.961530 9209 external_mini_cluster.cc:1428] Setting key 8d0d08ba33c783c62d0d28f760b9ebfa
Sep 24 01:02:57 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211777, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:02:57.971402 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:57.953027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:37775 (local address 127.8.254.126:38457)
0924 01:02:57.953499 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:57.953504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:57.954647 (+ 1143us) server_negotiation.cc:407] Connection header received
0924 01:02:57.956543 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:57.956551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:57.956663 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:57.956796 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:57.958403 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:57.959106 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:57.961105 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:57.961503 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:57.961646 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:57.965433 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:57.965444 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:57.965448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:57.965479 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:57.968069 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:57.968891 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:57.968895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:57.968898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:57.969018 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:57.969519 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:57.969523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:57.969525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:57.969797 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:57.969960 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:57.970795 (+ 835us) server_negotiation.cc:299] Negotiation successful
0924 01:02:57.971035 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260924 01:02:57.972427 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:57.953616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:37775)
0924 01:02:57.954453 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:57.954495 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:57.956258 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:57.957048 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:57.957057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:57.957591 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:57.958163 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:57.958181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:57.959274 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:57.959280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:57.960942 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:57.960957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:57.961759 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:57.961766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:57.962058 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:57.962872 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:57.962896 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:57.965198 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:57.968225 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:57.968235 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:57.968256 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:57.968740 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:57.969134 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:57.969138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:57.969142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:57.969370 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:57.969935 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:57.969953 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:57.970402 (+ 449us) client_negotiation.cc:769] Sending connection context
0924 01:02:57.970681 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 01:02:57.971257 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":309,"threads_started":1}
I20260924 01:02:57.974121 15610 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38457
I20260924 01:02:57.974506 15610 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:57.975421 15610 heartbeater.cc:511] Master 127.8.254.126:38457 requested a full tablet report, sending...
I20260924 01:02:57.978031 15421 ts_manager.cc:194] Registered new tserver with Master: 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283)
I20260924 01:02:57.980402 15421 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:37775
WARNING: no policy specified for oryx/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab.
Entry for principal oryx/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab.
I20260924 01:02:58.014072 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38457
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:58.343886 15616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:58.344172 15616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:58.344357 15616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:58.351627 15616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:58.351788 15616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:58.351962 15616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:02:58.361812 15616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:58.363214 15616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:58.364588 15616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:58.378376 15624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:58.378397 15621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:58.378428 15622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:58.379056 15616 server_base.cc:1034] running on GCE node
I20260924 01:02:58.379769 15616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:58.380846 15616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:58.382081 15616 hybrid_clock.cc:648] HybridClock initialized: now 1790211778382032 us; error 48 us; skew 500 ppm
I20260924 01:02:58.382457 15616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:58 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211778, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:58.385692 15616 init.cc:377] Logged in from keytab as oryx/127.8.254.66@KRBTEST.COM (short username oryx)
I20260924 01:02:58.387455 15616 webserver.cc:460] Webserver started at http://127.8.254.66:37341/ using document root <none> and password file <none>
I20260924 01:02:58.388190 15616 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:58.388329 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:58.388706 15616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:58.391820 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "a2b7ef6ee4f341db9e79fb6952e03a83"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "2f096966a7caa0b8afeb59af22f462fb"
server_key_iv: "201493323a0d45ce5edbae385ffec62e"
server_key_version: "encryptionkey@0"
I20260924 01:02:58.392624 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "a2b7ef6ee4f341db9e79fb6952e03a83"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "2f096966a7caa0b8afeb59af22f462fb"
server_key_iv: "201493323a0d45ce5edbae385ffec62e"
server_key_version: "encryptionkey@0"
I20260924 01:02:58.398403 15616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 01:02:58.402298 15631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:58.403801 15616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:02:58.403996 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "a2b7ef6ee4f341db9e79fb6952e03a83"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "2f096966a7caa0b8afeb59af22f462fb"
server_key_iv: "201493323a0d45ce5edbae385ffec62e"
server_key_version: "encryptionkey@0"
I20260924 01:02:58.404220 15616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:58.434538 15616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:58.438733 15616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:58.439003 15616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:58.439947 15616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:58.441251 15616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:58.441327 15616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:58.441416 15616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:58.441476 15616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:58.476373 15616 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:41739
I20260924 01:02:58.476523 15743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:41739 every 8 connection(s)
I20260924 01:02:58.477746 15616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:02:58.486155 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15616
I20260924 01:02:58.486492 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:02:58.486960 9209 external_mini_cluster.cc:1428] Setting key 0523434c8de08a9285c1738508de48d1
Sep 24 01:02:58 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211778, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:02:58.498548 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:58.480412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54517 (local address 127.8.254.126:38457)
0924 01:02:58.480900 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:58.480906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:58.481967 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 01:02:58.483841 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:58.483845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:58.483912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:58.484012 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:58.485495 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.486244 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.488090 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.488605 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.488773 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:58.492535 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:58.492541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:58.492544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:58.492576 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:58.495457 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:58.496193 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:58.496199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:58.496202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:58.496335 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:58.496772 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:58.496780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:58.496783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:58.497051 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:58.497222 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:58.497894 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 01:02:58.498115 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260924 01:02:58.499547 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:58.481058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:54517)
0924 01:02:58.481785 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:58.481824 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:58.483589 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:58.484214 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:58.484222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:58.484775 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:58.485338 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:58.485355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.486411 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.486415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:58.487929 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:58.487950 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.488870 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.488879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:58.489124 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:58.489828 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:58.489852 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:58.492287 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:58.495605 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:58.495614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:58.495631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:58.496065 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:58.496450 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:58.496457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:58.496461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:58.496655 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:58.497183 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:58.497192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:58.497542 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 01:02:58.497819 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 01:02:58.498395 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":253,"threads_started":1}
I20260924 01:02:58.501083 15744 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38457
I20260924 01:02:58.501400 15744 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:58.502112 15744 heartbeater.cc:511] Master 127.8.254.126:38457 requested a full tablet report, sending...
I20260924 01:02:58.503751 15421 ts_manager.cc:194] Registered new tserver with Master: a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66:41739)
I20260924 01:02:58.505355 15421 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:54517
WARNING: no policy specified for oryx/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab.
Entry for principal oryx/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab.
I20260924 01:02:58.537990 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38457
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:58.867681 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:58.867965 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:58.868059 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:58.875519 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:58.875639 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:58.875754 15750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:02:58.885771 15750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:02:58.887104 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:58.888484 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:58.901695 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:58.901719 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:58.901898 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:58.902731 15750 server_base.cc:1034] running on GCE node
I20260924 01:02:58.903417 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:58.904369 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:58.905599 15750 hybrid_clock.cc:648] HybridClock initialized: now 1790211778905522 us; error 74 us; skew 500 ppm
I20260924 01:02:58.905930 15750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:58 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211778, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:58.909333 15750 init.cc:377] Logged in from keytab as oryx/127.8.254.67@KRBTEST.COM (short username oryx)
I20260924 01:02:58.910975 15750 webserver.cc:460] Webserver started at http://127.8.254.67:42839/ using document root <none> and password file <none>
I20260924 01:02:58.911702 15750 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:58.911849 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:58.912163 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:58.915282 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "3fef41032fb241dd8d8c1bcdf5d09396"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "d4877aa394868b131b0eb2ab9bd3bf02"
server_key_iv: "b68dd1ccfa164e9315db497d1d1fd581"
server_key_version: "encryptionkey@0"
I20260924 01:02:58.916087 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "3fef41032fb241dd8d8c1bcdf5d09396"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "d4877aa394868b131b0eb2ab9bd3bf02"
server_key_iv: "b68dd1ccfa164e9315db497d1d1fd581"
server_key_version: "encryptionkey@0"
I20260924 01:02:58.921319 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:02:58.924875 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:58.926263 15750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:02:58.926467 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "3fef41032fb241dd8d8c1bcdf5d09396"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "d4877aa394868b131b0eb2ab9bd3bf02"
server_key_iv: "b68dd1ccfa164e9315db497d1d1fd581"
server_key_version: "encryptionkey@0"
I20260924 01:02:58.926687 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:58.942732 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:58.946642 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:58.946960 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:58.947978 15750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:02:58.949707 15750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:02:58.949836 15750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:58.949957 15750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:02:58.950021 15750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:58.984539 15610 heartbeater.cc:503] Master 127.8.254.126:38457 was elected leader, sending a full tablet report...
I20260924 01:02:58.985146 15750 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:32783
I20260924 01:02:58.985175 15877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:32783 every 8 connection(s)
I20260924 01:02:58.986878 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:02:58.988324 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15750
I20260924 01:02:58.988533 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:02:58.989228 9209 external_mini_cluster.cc:1428] Setting key fead5089beaca13931249881b1f99528
Sep 24 01:02:58 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211778, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:02:59.007752 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:58.989929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:57223 (local address 127.8.254.126:38457)
0924 01:02:58.990139 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:58.990145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:58.991520 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 01:02:58.993390 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:58.993394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:58.993475 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:58.993560 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:58.995201 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.995991 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.997509 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.997907 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.998014 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.001676 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.001681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.001685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.001722 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.004487 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.005270 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.005276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.005278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.005405 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.005872 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.005881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.005885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.006167 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.006399 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.007151 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.007416 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:02:59.008846 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:58.990519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:57223)
0924 01:02:58.991383 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:58.991430 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:58.993124 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:58.993773 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:58.993781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:58.994449 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:58.995035 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:58.995054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.996124 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.996128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:58.997373 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:58.997385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:58.998104 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:58.998109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:58.998384 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:58.999010 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:58.999033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.001436 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.004645 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.004655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.004675 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.005132 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.005521 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.005526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.005530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.005748 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.006360 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.006369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.006761 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.007053 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.007621 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":346,"threads_started":1}
I20260924 01:02:59.010371 15878 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38457
I20260924 01:02:59.010679 15878 heartbeater.cc:464] Registering TS with master...
I20260924 01:02:59.011400 15878 heartbeater.cc:511] Master 127.8.254.126:38457 requested a full tablet report, sending...
I20260924 01:02:59.012833 15421 ts_manager.cc:194] Registered new tserver with Master: 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:02:59.013701 15421 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:57223
I20260924 01:02:59.017163 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.254.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.125.keytab.
Entry for principal oryx/127.8.254.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.125.keytab.
I20260924 01:02:59.070436 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:36617
--webserver_interface=127.8.254.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.125.keytab
--principal=oryx/127.8.254.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38457,127.8.254.125:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:02:59.400451 15883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:02:59.400743 15883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:02:59.400924 15883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:02:59.408075 15883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:02:59.408226 15883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:02:59.408317 15883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:02:59.408383 15883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:02:59.408455 15883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:02:59.418744 15883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38457,127.8.254.125:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.125.keytab
--principal=oryx/127.8.254.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.254.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
W20260924 01:02:59.419129 15883 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.254.126:38457,127.8.254.125:36617'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 01:02:59.420179 15883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:02:59.421463 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:59.433743 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:59.433785 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:59.433826 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:59.434744 15883 server_base.cc:1034] running on GCE node
I20260924 01:02:59.435549 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:59.437099 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:59.438314 15883 hybrid_clock.cc:648] HybridClock initialized: now 1790211779438251 us; error 36 us; skew 500 ppm
I20260924 01:02:59.438683 15883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:59 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211779, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:59.441871 15883 init.cc:377] Logged in from keytab as oryx/127.8.254.125@KRBTEST.COM (short username oryx)
I20260924 01:02:59.443418 15883 webserver.cc:460] Webserver started at http://127.8.254.125:40705/ using document root <none> and password file <none>
I20260924 01:02:59.444094 15883 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:59.444183 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:59.444440 15883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:02:59.447566 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/instance:
uuid: "00805083ae6244a0876a5fda23e25e1d"
format_stamp: "Formatted at 2026-09-24 01:02:59 on dist-test-slave-5c7l"
server_key: "20af29b8bc09506f9c6391b9ee938da1"
server_key_iv: "c977bf352fe7e1d2ad969cd0bae40d0d"
server_key_version: "encryptionkey@0"
I20260924 01:02:59.448295 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal/instance:
uuid: "00805083ae6244a0876a5fda23e25e1d"
format_stamp: "Formatted at 2026-09-24 01:02:59 on dist-test-slave-5c7l"
server_key: "20af29b8bc09506f9c6391b9ee938da1"
server_key_iv: "c977bf352fe7e1d2ad969cd0bae40d0d"
server_key_version: "encryptionkey@0"
I20260924 01:02:59.453588 15883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:02:59.457098 15898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:59.458600 15883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 01:02:59.458789 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
uuid: "00805083ae6244a0876a5fda23e25e1d"
format_stamp: "Formatted at 2026-09-24 01:02:59 on dist-test-slave-5c7l"
server_key: "20af29b8bc09506f9c6391b9ee938da1"
server_key_iv: "c977bf352fe7e1d2ad969cd0bae40d0d"
server_key_version: "encryptionkey@0"
I20260924 01:02:59.459029 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:59.480065 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:59.484520 15883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:02:59.484799 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:59.506531 15883 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.125:36617
I20260924 01:02:59.506543 15949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.125:36617 every 8 connection(s)
I20260924 01:02:59.508230 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
I20260924 01:02:59.508203 15744 heartbeater.cc:503] Master 127.8.254.126:38457 was elected leader, sending a full tablet report...
I20260924 01:02:59.511200 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15883
I20260924 01:02:59.513172 15950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:02:59.522181 15950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:02:59 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
Sep 24 01:02:59 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211779, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.125@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:02:59.536015 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.513496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.8.254.125:36617)
0924 01:02:59.514329 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.514342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.514383 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:02:59.515968 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.515990 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.516388 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.516754 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.518572 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.519918 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.521640 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.522389 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.522614 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.527553 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.527563 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.527575 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.527616 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.531411 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.532312 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.532318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.532347 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.532596 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.533064 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.533072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.533077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.533659 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.534009 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.534516 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.534921 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":359,"threads_started":1}
I20260924 01:02:59.541021 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.525403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.8.254.126:38457)
0924 01:02:59.526020 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.526027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.526553 (+ 526us) server_negotiation.cc:407] Connection header received
0924 01:02:59.526752 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.526760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.526866 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.526993 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.528458 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.529356 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.531572 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.532262 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.532378 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.535776 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.535787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.535791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.535826 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.538519 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.539258 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.539262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.539265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.539383 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.539746 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.539749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.539751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.540079 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.540193 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.540614 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.540779 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":281,"threads_started":1}
I20260924 01:02:59.541023 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.525374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:38418)
0924 01:02:59.526400 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:59.526458 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:59.526611 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:59.527055 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:59.527060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:59.527911 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:59.528308 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.528320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.529524 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.529531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.531403 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.531413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.532467 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.532477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.532660 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.533312 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:59.533328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.535579 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.538676 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.538685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.538690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.539139 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.539507 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.539511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.539514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.539644 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.540199 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.540205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.540354 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.540510 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.540735 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":624,"threads_started":1}
I20260924 01:02:59.542785 15950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:02:59 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211779, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.125@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
I20260924 01:02:59.554713 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.543555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:50512)
0924 01:02:59.543764 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:59.543784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:59.543894 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:59.544156 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:59.544159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:59.544471 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:59.544599 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.544607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.545550 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.545554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.546554 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.546567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.547222 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.547228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.547354 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.547864 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:59.547880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.549794 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.552610 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.552614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.552617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.552975 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.553357 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.553360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.553363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.553490 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.553932 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.553935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.554054 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.554160 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.554472 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 01:02:59.554932 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.543805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.8.254.125:36617)
0924 01:02:59.543937 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.543941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.543959 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:02:59.543996 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.544000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.544048 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.544122 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.544712 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.545403 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.546687 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.547076 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.547164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.549962 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.549966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.549970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.549997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.552485 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.553076 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.553079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.553081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.553222 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.553588 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.553591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.553593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.553832 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.553936 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.554298 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.554606 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":48}
I20260924 01:02:59.559542 15950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Bootstrap starting.
I20260924 01:02:59.562855 15950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Neither blocks nor log segments found. Creating new log.
I20260924 01:02:59.564158 15950 log.cc:824] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Log is configured to *not* fsync() on all Append() calls
I20260924 01:02:59.567777 15950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: No bootstrap required, opened a new log
I20260924 01:02:59.574848 15950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } }
I20260924 01:02:59.575156 15950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:02:59.575273 15950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00805083ae6244a0876a5fda23e25e1d, State: Initialized, Role: FOLLOWER
I20260924 01:02:59.575870 15950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } }
I20260924 01:02:59.576689 15957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } } }
I20260924 01:02:59.577018 15957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:02:59.577744 15950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: configured and running, proceeding with master startup.
W20260924 01:02:59.586783 15968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:02:59.586930 15968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:02:59.588797 15883 master_runner.cc:428] Detected that this master 00805083ae6244a0876a5fda23e25e1d is joining an existing cluster
I20260924 01:02:59.588894 15883 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.254.125:36617
WARNING: no policy specified for oryx/127.8.254.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.124@KRBTEST.COM" created.
I20260924 01:02:59.625393 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.597080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.8.254.125:36617)
0924 01:02:59.597561 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.597569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.609746 (+ 12177us) server_negotiation.cc:407] Connection header received
0924 01:02:59.609972 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.609978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.610041 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.610114 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.611516 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.612647 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.614674 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.615420 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.615521 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.618013 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.618021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.618026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.618065 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.622040 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.622943 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.622951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.622957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.623119 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.623566 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.623574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.623580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.624006 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.624169 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.624650 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.624909 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":308}
I20260924 01:02:59.626485 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.596529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:50520)
0924 01:02:59.609639 (+ 13110us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:59.609665 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:59.609843 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:59.610357 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:59.610364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:59.611014 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:59.611347 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.611375 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.612865 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.612870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.614521 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.614537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.615647 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.615653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.615824 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.616564 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:59.616587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.617769 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.622217 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.622223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.622227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.622760 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.623251 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.623256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.623260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.623430 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.624220 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.624228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.624396 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.625605 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.625902 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12635,"mutex_wait_us":1,"thread_start_us":10347,"threads_started":1}
I20260924 01:02:59.629858 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.596992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.8.254.126:38457)
0924 01:02:59.597354 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.597365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.607430 (+ 10065us) server_negotiation.cc:407] Connection header received
0924 01:02:59.607802 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.607811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.607956 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.608040 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.610046 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.611358 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.613685 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.614467 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.614561 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.617584 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.617589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.617593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.617628 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.626548 (+ 8920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.627570 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.627577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.627580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.627740 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.628220 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.628226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.628229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.628625 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.628786 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.629448 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.629636 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260924 01:02:59.630106 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.596467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:38426)
0924 01:02:59.607283 (+ 10816us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:59.607310 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:59.607634 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:59.608306 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:59.608314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:59.609212 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:59.609827 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.609862 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.611578 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.611587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.613503 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.613522 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.614901 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.614920 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.615730 (+ 810us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.616397 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:59.616423 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.617314 (+ 891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.626887 (+ 9573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.626892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.626895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.627425 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.627899 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.627904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.627908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.628077 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.628905 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.628910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.629134 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.629295 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.629647 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10602,"mutex_wait_us":3,"thread_start_us":12764,"threads_started":1}
Sep 24 01:02:59 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211777, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.126@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
I20260924 01:02:59.651070 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.636046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.8.254.125:36617)
0924 01:02:59.636221 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.636227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.636495 (+ 268us) server_negotiation.cc:407] Connection header received
0924 01:02:59.636861 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.636866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.636987 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.637044 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.638290 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.639034 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.640400 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.640914 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.640996 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.644480 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.644485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.644488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.644521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.647719 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.648554 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.648563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.648567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.648711 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.649146 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.649152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.649156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.649555 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.649717 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.650443 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.650676 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:02:59.651120 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.635743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:50530)
0924 01:02:59.636367 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:59.636416 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:59.636623 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:59.637253 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:59.637259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:59.637900 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:59.638117 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.638126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.639169 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.639175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.640218 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.640235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.641208 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.641212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.641351 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.641998 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:59.642017 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.644260 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.647858 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.647877 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.647886 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.648424 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.648839 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.648846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.648852 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.649032 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.649689 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.649698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.649995 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.650350 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.650693 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":488,"threads_started":1}
I20260924 01:02:59.653225 15421 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.254.125:36617: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "00805083ae6244a0876a5fda23e25e1d"
member_type: NON_VOTER
last_known_addr {
host: "127.8.254.125"
port: 36617
}
attrs {
promote: true
}
}
dest_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.124.keytab.
Entry for principal oryx/127.8.254.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.124.keytab.
I20260924 01:02:59.654433 15421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } }
I20260924 01:02:59.656207 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:40533
--webserver_interface=127.8.254.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.124.keytab
--principal=oryx/127.8.254.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:02:59.668671 15925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:02:59.672771 15925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Refusing update from remote peer 1bd51e6cfd7d4588a8ebdfe215842ac7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 01:02:59.673182 15981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 00805083ae6244a0876a5fda23e25e1d (127.8.254.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.674050 15980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:02:59.695173 15981 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.254.125:36617
I20260924 01:02:59.696094 15883 master.cc:537] Master@127.8.254.125:36617 shutting down...
I20260924 01:02:59.697124 15980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.697503 15980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:02:59.709899 15925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 00805083ae6244a0876a5fda23e25e1d (127.8.254.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.730737 15991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.731267 15986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.731657 15986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: LEARNER
I20260924 01:02:59.731510 15957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51e6cfd7d4588a8ebdfe215842ac7. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.731763 15957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: LEARNER
I20260924 01:02:59.731258 15991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: LEARNER
I20260924 01:02:59.734653 15883 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 LEARNER]: Raft consensus shutting down.
I20260924 01:02:59.735173 15883 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 LEARNER]: Raft consensus is shut down!
I20260924 01:02:59.735325 15883 tablet_replica.cc:330] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: stopping tablet replica
I20260924 01:02:59.743295 15883 master.cc:552] Master@127.8.254.125:36617 shutdown complete.
I20260924 01:02:59.743466 15883 master_runner.cc:305] Clearing existing system tablet
I20260924 01:02:59.743971 15883 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:59.744143 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:59.747171 15883 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 01:02:59.749089 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:59.749461 15883 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:59.749877 15883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 01:02:59.750011 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
uuid: "00805083ae6244a0876a5fda23e25e1d"
format_stamp: "Formatted at 2026-09-24 01:02:59 on dist-test-slave-5c7l"
server_key: "20af29b8bc09506f9c6391b9ee938da1"
server_key_iv: "c977bf352fe7e1d2ad969cd0bae40d0d"
server_key_version: "encryptionkey@0"
I20260924 01:02:59.754709 15883 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:02:59.762048 15883 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 01:02:59.762465 15883 log.cc:1196] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 01:02:59.763053 15883 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Deleting consensus metadata
I20260924 01:02:59.763368 15883 master_runner.cc:315] Copying system tablet from 127.8.254.126:38457
I20260924 01:02:59.766098 15883 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.254.126:38457
I20260924 01:02:59.786106 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.767083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.8.254.126:38457)
0924 01:02:59.767293 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:02:59.767300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:02:59.770797 (+ 3497us) server_negotiation.cc:407] Connection header received
0924 01:02:59.771127 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:02:59.771134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:02:59.771225 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:02:59.771347 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:02:59.772398 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.773556 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.775503 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.776180 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.776324 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.778946 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:02:59.778954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:02:59.778959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:02:59.779031 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:02:59.782695 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.783564 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.783570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.783574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.783739 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:02:59.784246 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:02:59.784254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:02:59.784257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:02:59.784650 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:02:59.784809 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 01:02:59.785437 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 01:02:59.785704 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 01:02:59.786293 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:02:59.767052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:38432)
0924 01:02:59.770652 (+ 3600us) negotiation.cc:107] Waiting for socket to connect
0924 01:02:59.770680 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 01:02:59.770931 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:02:59.771396 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:02:59.771401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:02:59.771915 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:02:59.772215 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.772234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.773745 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.773750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.775333 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:02:59.775351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:02:59.776368 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:02:59.776375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:02:59.776612 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:02:59.777465 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:02:59.777493 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:02:59.778727 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:02:59.782879 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.782885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.782889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.783415 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.783885 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:02:59.783892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:02:59.783897 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:02:59.784100 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:02:59.784805 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:02:59.784812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:02:59.785102 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 01:02:59.785280 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 01:02:59.785645 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3450,"thread_start_us":3302,"threads_started":1}
I20260924 01:02:59.787528 15441 tablet_copy_service.cc:140] P 1bd51e6cfd7d4588a8ebdfe215842ac7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 00805083ae6244a0876a5fda23e25e1d ({username='oryx', principal='oryx/127.8.254.125@KRBTEST.COM'} at 127.0.0.1:38432)
I20260924 01:02:59.787890 15441 tablet_copy_service.cc:161] P 1bd51e6cfd7d4588a8ebdfe215842ac7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 00805083ae6244a0876a5fda23e25e1d at {username='oryx', principal='oryx/127.8.254.125@KRBTEST.COM'} at 127.0.0.1:38432: session id = 00805083ae6244a0876a5fda23e25e1d-00000000000000000000000000000000
I20260924 01:02:59.792584 15441 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 01:02:59.795430 15883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 01:02:59.802655 15883 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: tablet copy: Starting download of 0 data blocks...
I20260924 01:02:59.803002 15883 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: tablet copy: Starting download of 1 WAL segments...
I20260924 01:02:59.806061 15883 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 01:02:59.810675 15441 tablet_copy_service.cc:342] P 1bd51e6cfd7d4588a8ebdfe215842ac7: Request end of tablet copy session 00805083ae6244a0876a5fda23e25e1d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.254.125@KRBTEST.COM'} at 127.0.0.1:38432
I20260924 01:02:59.810904 15441 tablet_copy_service.cc:433] P 1bd51e6cfd7d4588a8ebdfe215842ac7: ending tablet copy session 00805083ae6244a0876a5fda23e25e1d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 00805083ae6244a0876a5fda23e25e1d
I20260924 01:02:59.824369 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:02:59.827381 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:59.831661 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:02:59.847849 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:02:59.855914 15883 server_base.cc:1034] running on GCE node
I20260924 01:02:59.856333 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:02:59.857081 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:02:59.858418 15883 hybrid_clock.cc:648] HybridClock initialized: now 1790211779858377 us; error 79 us; skew 500 ppm
I20260924 01:02:59.858728 15883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:02:59 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211779, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:02:59.861179 15883 init.cc:377] Logged in from keytab as oryx/127.8.254.125@KRBTEST.COM (short username oryx)
I20260924 01:02:59.863230 15883 webserver.cc:460] Webserver started at http://127.8.254.125:33245/ using document root <none> and password file <none>
I20260924 01:02:59.863621 15883 fs_manager.cc:359] Metadata directory not provided
I20260924 01:02:59.863709 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:02:59.867090 15883 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 01:02:59.869184 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:02:59.869663 15883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 01:02:59.869809 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
uuid: "00805083ae6244a0876a5fda23e25e1d"
format_stamp: "Formatted at 2026-09-24 01:02:59 on dist-test-slave-5c7l"
server_key: "20af29b8bc09506f9c6391b9ee938da1"
server_key_iv: "c977bf352fe7e1d2ad969cd0bae40d0d"
server_key_version: "encryptionkey@0"
I20260924 01:02:59.869999 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:02:59.888926 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:02:59.893500 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:02:59.943508 15883 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.125:36617
I20260924 01:02:59.943521 16061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.125:36617 every 8 connection(s)
I20260924 01:02:59.944679 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
I20260924 01:02:59.950631 16062 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:02:59.955700 16062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Bootstrap starting.
I20260924 01:02:59.975606 16062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:02:59.976265 16062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: Bootstrap complete.
I20260924 01:02:59.977494 16062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } }
I20260924 01:02:59.977778 16062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00805083ae6244a0876a5fda23e25e1d, State: Initialized, Role: LEARNER
I20260924 01:02:59.978080 16062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } }
I20260924 01:02:59.980140 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:02:59.980554 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: LEARNER
I20260924 01:02:59.980635 16062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:00.001062 16076 catalog_manager.cc:1259] Loaded cluster ID: aafcc22a693344fc831348d8a2da7bdb
I20260924 01:03:00.001191 16076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: loading cluster ID for follower catalog manager: success
I20260924 01:03:00.005265 16076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: acquiring CA information for follower catalog manager: success
I20260924 01:03:00.008960 16076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 01:03:00.016358 15878 heartbeater.cc:503] Master 127.8.254.126:38457 was elected leader, sending a full tablet report...
W20260924 01:03:00.090305 15982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:00.090590 15982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:00.090749 15982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:00.099150 15982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:00.099257 15982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:00.099360 15982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:00.099460 15982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:00.099521 15982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:00.109645 15982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.124.keytab
--principal=oryx/127.8.254.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.254.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:00.111054 15982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:00.112481 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:00.126156 16082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:00.126137 16081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:00.127019 16084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:00.127693 15982 server_base.cc:1034] running on GCE node
I20260924 01:03:00.128520 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:00.130079 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:00.131301 15982 hybrid_clock.cc:648] HybridClock initialized: now 1790211780131262 us; error 33 us; skew 500 ppm
I20260924 01:03:00.131736 15982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211780, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:00.135365 15982 init.cc:377] Logged in from keytab as oryx/127.8.254.124@KRBTEST.COM (short username oryx)
I20260924 01:03:00.136920 15982 webserver.cc:460] Webserver started at http://127.8.254.124:39671/ using document root <none> and password file <none>
I20260924 01:03:00.137638 15982 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:00.137755 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:00.138027 15982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:00.141314 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/instance:
uuid: "2130f3384151439b885c797d8eb9cc7f"
format_stamp: "Formatted at 2026-09-24 01:03:00 on dist-test-slave-5c7l"
server_key: "eb76c03a61394060ad92977a03e5ebfc"
server_key_iv: "6653c35dc08e6589972bb336e1f75e52"
server_key_version: "encryptionkey@0"
I20260924 01:03:00.142124 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal/instance:
uuid: "2130f3384151439b885c797d8eb9cc7f"
format_stamp: "Formatted at 2026-09-24 01:03:00 on dist-test-slave-5c7l"
server_key: "eb76c03a61394060ad92977a03e5ebfc"
server_key_iv: "6653c35dc08e6589972bb336e1f75e52"
server_key_version: "encryptionkey@0"
I20260924 01:03:00.147709 15982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 01:03:00.151324 16091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:00.152805 15982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 01:03:00.153004 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
uuid: "2130f3384151439b885c797d8eb9cc7f"
format_stamp: "Formatted at 2026-09-24 01:03:00 on dist-test-slave-5c7l"
server_key: "eb76c03a61394060ad92977a03e5ebfc"
server_key_iv: "6653c35dc08e6589972bb336e1f75e52"
server_key_version: "encryptionkey@0"
I20260924 01:03:00.153236 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:00.169636 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:00.174088 15982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:00.174430 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:00.194576 15982 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.124:40533
I20260924 01:03:00.194583 16142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.124:40533 every 8 connection(s)
I20260924 01:03:00.196213 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
I20260924 01:03:00.199908 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:00.204264 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 15982
I20260924 01:03:00.208109 16143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.124@KRBTEST.COM
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211780, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.124@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:00.228300 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.211719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.8.254.126:38457)
0924 01:03:00.211951 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.211957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.212582 (+ 625us) server_negotiation.cc:407] Connection header received
0924 01:03:00.212753 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.212758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.212838 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.212948 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.214477 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.215247 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.217060 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.217443 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.217557 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.221792 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.221800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.221804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.221844 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.225166 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.225878 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.225884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.225889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.226028 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.226545 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.226550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.226553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.226868 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.227023 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.227711 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.227966 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 01:03:00.229492 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.206492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.8.254.124:40533)
0924 01:03:00.207246 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.207258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.207299 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:03:00.209108 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.209141 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.209629 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.210043 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.211676 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.213443 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.215043 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.215966 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.216097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.221499 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.221508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.221522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.221557 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.225380 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.226060 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.226069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.226072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.226331 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.226748 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.226753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.226756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.227338 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.227696 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.228051 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.228341 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":559,"spinlock_wait_cycles":17152,"thread_start_us":350,"threads_started":1}
I20260924 01:03:00.229526 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.211418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:38442)
0924 01:03:00.212426 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.212478 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.212630 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.213008 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.213013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.213938 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.214297 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.214311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.215396 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.215400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.216910 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.216928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.217694 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.217703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.217908 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.218571 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.218608 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.221587 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.225305 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.225314 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.225325 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.225738 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.226145 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.226150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.226156 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.226430 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.226996 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.227005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.227297 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.227615 (+ 318us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.228167 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":593,"threads_started":1}
I20260924 01:03:00.232139 16143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:03:00.237715 16149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:03:00.238008 16149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211780, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.124@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
I20260924 01:03:00.250399 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.233005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:50542)
0924 01:03:00.233226 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.233242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.233326 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.234008 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.234011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.234410 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.234576 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.234585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.235876 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.235884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.237021 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.237034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.238242 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.238248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.238528 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.239538 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.239572 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.243259 (+ 3687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.247839 (+ 4580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.247844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.247849 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.248407 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.248773 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.248778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.248782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.248953 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.249425 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.249430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.249582 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.249730 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.250064 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 01:03:00.250514 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.233053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.8.254.125:36617)
0924 01:03:00.233716 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.233720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.233766 (+ 46us) server_negotiation.cc:407] Connection header received
0924 01:03:00.233818 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.233822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.233880 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.233964 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.234726 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.235724 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.237165 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.237832 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.237960 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.243600 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.243610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.243618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.243674 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.247694 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.248522 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.248527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.248529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.248658 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.249058 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.249061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.249063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.249311 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.249436 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.249808 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.250026 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":383,"threads_started":1}
I20260924 01:03:00.251932 16143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:03:00.252523 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.237910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:50554)
0924 01:03:00.245836 (+ 7926us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.245860 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.246012 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.246485 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.246490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.246517 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:00.246793 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.246805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.247957 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.247964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.249886 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.249902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.251681 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.251687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.251870 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.251928 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.252041 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.252181 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7710,"thread_start_us":7522,"threads_started":1}
I20260924 01:03:00.252519 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.238421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.8.254.125:36617)
0924 01:03:00.245276 (+ 6855us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.245285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.246090 (+ 805us) server_negotiation.cc:407] Connection header received
0924 01:03:00.246152 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.246156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.246306 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.246409 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:00.246939 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.247812 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.250051 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.251143 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.251264 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.251470 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.252065 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.252193 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6680,"thread_start_us":6497,"threads_started":1}
I20260924 01:03:00.253676 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51e6cfd7d4588a8ebdfe215842ac7. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: NON_VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: true } } }
I20260924 01:03:00.253944 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: LEARNER
I20260924 01:03:00.254552 16150 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: attempting to promote NON_VOTER 00805083ae6244a0876a5fda23e25e1d to VOTER
I20260924 01:03:00.255468 16150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } }
I20260924 01:03:00.257375 16037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 LEARNER]: Refusing update from remote peer 1bd51e6cfd7d4588a8ebdfe215842ac7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 01:03:00.257995 16149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211780, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.124@KRBTEST.COM for oryx/127.8.254.124@KRBTEST.COM
I20260924 01:03:00.261135 16150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 00805083ae6244a0876a5fda23e25e1d (127.8.254.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } }
I20260924 01:03:00.261889 16037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 00805083ae6244a0876a5fda23e25e1d (127.8.254.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } }
I20260924 01:03:00.264348 16150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } }
I20260924 01:03:00.264639 16150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:00.264564 16149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } }
I20260924 01:03:00.264936 16149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:00.266018 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } }
I20260924 01:03:00.266273 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:03:00.267843 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.252836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.0.0.1:50862)
0924 01:03:00.253079 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.253094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.253184 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.253528 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.253533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.253997 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.254175 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.254184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.255268 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.255273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.256539 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.256557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.257446 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.257455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.257648 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.258377 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.258401 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.260933 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.264834 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.264839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.264844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.265331 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.265765 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.265772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.265776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.266002 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.266715 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.266720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.266893 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.267046 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.267352 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 01:03:00.267869 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.252843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.8.254.124:40533)
0924 01:03:00.253012 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.253018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.253162 (+ 144us) server_negotiation.cc:407] Connection header received
0924 01:03:00.253279 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.253283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.253357 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.253457 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.254345 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.255108 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.256691 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.257162 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.257277 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.261117 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.261124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.261129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.261167 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.264584 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.265488 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.265492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.265495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.265618 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.266131 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.266137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.266141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.266579 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.266744 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.267127 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.267492 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":60}
I20260924 01:03:00.272994 16143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Bootstrap starting.
I20260924 01:03:00.276497 16143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:00.277627 16143 log.cc:824] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:00.280938 16143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: No bootstrap required, opened a new log
I20260924 01:03:00.289510 16143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } }
I20260924 01:03:00.289875 16143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:00.289959 16143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2130f3384151439b885c797d8eb9cc7f, State: Initialized, Role: FOLLOWER
I20260924 01:03:00.290645 16143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } }
I20260924 01:03:00.291575 16157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } } }
I20260924 01:03:00.291937 16157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:03:00.292671 16143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 01:03:00.303180 16168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:03:00.303344 16168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:03:00.307475 15982 master_runner.cc:428] Detected that this master 2130f3384151439b885c797d8eb9cc7f is joining an existing cluster
I20260924 01:03:00.307646 15982 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.254.124:40533
I20260924 01:03:00.329869 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.312501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.8.254.125:36617)
0924 01:03:00.312720 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.312729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.313448 (+ 719us) server_negotiation.cc:407] Connection header received
0924 01:03:00.313679 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.313685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.313786 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.313905 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.315569 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.316728 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.319029 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.319693 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.319814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.322616 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.322623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.322628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.322668 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.326365 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.327190 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.327197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.327200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.327372 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.327968 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.327973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.327978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.328328 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.328468 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.329120 (+ 652us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.329481 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:03:00.332125 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.311828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:50570)
0924 01:03:00.313311 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.313333 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.313514 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.313952 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.313959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.314900 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.315370 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.315387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.316915 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.316921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.318821 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.318843 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.320143 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.320150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.320320 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.321106 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.321130 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.322388 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.326538 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.326548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.326552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.327036 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.327542 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.327551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.327556 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.327826 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.328548 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.328553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.328772 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.330606 (+ 1834us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.331044 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":11296,"threads_started":1}
I20260924 01:03:00.336592 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15482
I20260924 01:03:00.342170 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.311973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.8.254.124:40533)
0924 01:03:00.312170 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.312176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.322950 (+ 10774us) server_negotiation.cc:407] Connection header received
0924 01:03:00.323870 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.323878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.323940 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.323999 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.325849 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.326968 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.328990 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.329650 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.329738 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.333041 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.333048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.333053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.333101 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.338046 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.339067 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.339072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.339081 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.339294 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.339815 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.339821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.339824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.340435 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.340576 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.341298 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.341614 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:03:00.343649 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.311854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.0.0.1:50866)
0924 01:03:00.322829 (+ 10975us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.322857 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.323752 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.324269 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.324275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.325178 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.325689 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.325703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.327207 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.327212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.328842 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.328856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.329989 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.329994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.330148 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.330923 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.330945 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.332803 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.338238 (+ 5435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.338243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.338247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.338932 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.339486 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.339491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.339494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.339683 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.340668 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.340673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.340917 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.342459 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.342905 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10765,"mutex_wait_us":18,"thread_start_us":10555,"threads_started":1}
I20260924 01:03:00.349498 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.312318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:38458)
0924 01:03:00.323410 (+ 11092us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.323424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.323578 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.324128 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.324132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.325163 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.325652 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.325673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.327234 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.327241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.329134 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.329147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.330498 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.330506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.330716 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.332163 (+ 1447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.332188 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.333435 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.346391 (+ 12956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.346396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.346401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.346972 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.347433 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.347438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.347442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.347602 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.348351 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.348357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.348581 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.348708 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.349037 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10959,"mutex_wait_us":22,"thread_start_us":624,"threads_started":1}
I20260924 01:03:00.350508 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.311605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.8.254.126:38457)
0924 01:03:00.311814 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.311822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.323545 (+ 11723us) server_negotiation.cc:407] Connection header received
0924 01:03:00.323725 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.323730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.323823 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.323916 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.325871 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.327047 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.329316 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.330002 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.330104 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.333698 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.333706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.333711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.333751 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.342622 (+ 8871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.347103 (+ 4481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.347110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.347114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.347286 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.347746 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.347752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.347755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.348138 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.348281 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.349867 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.350108 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.65@KRBTEST.COM
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211777, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.126@KRBTEST.COM for oryx/127.8.254.124@KRBTEST.COM
I20260924 01:03:00.368832 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.338615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.8.254.65:40283)
0924 01:03:00.347592 (+ 8977us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.347602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.347633 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:00.347705 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.347716 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.348075 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.350084 (+ 2009us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.350405 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.352018 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.354007 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.354965 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.355742 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.361147 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.361163 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.361186 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.361242 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.364968 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.365890 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.365902 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.365907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.366083 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.366608 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.366616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.366622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.367113 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.367435 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.367867 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.368161 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8769,"thread_start_us":20445,"threads_started":1}
I20260924 01:03:00.370464 15605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:00.373344 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.356225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.0.0.1:50882)
0924 01:03:00.356445 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.356467 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.356578 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.357476 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.357479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.357918 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.358095 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.358104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.359160 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.359166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.360644 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.360661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.361559 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.361565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.361750 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.362376 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.362402 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.364786 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.368789 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.368794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.368801 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.369402 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.369830 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.369837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.369841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.370019 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.370858 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.370866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.371056 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.371268 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.372372 (+ 1104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 01:03:00.373380 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.356745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.8.254.124:40533)
0924 01:03:00.356923 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.356927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.356944 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:03:00.357021 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.357024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.357108 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.357243 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.358285 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.359019 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.360839 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.361399 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.361547 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.365021 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.365030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.365036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.365082 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.368606 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.369535 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.369542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.369545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.369689 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.370139 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.370145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.370148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.370628 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.370790 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.371830 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.372907 (+ 1077us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 01:03:00.375015 15421 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.254.124:40533: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2130f3384151439b885c797d8eb9cc7f"
member_type: NON_VOTER
last_known_addr {
host: "127.8.254.124"
port: 40533
}
attrs {
promote: true
}
}
dest_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7"
cas_config_opid_index: 6
I20260924 01:03:00.375787 15421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } }
I20260924 01:03:00.378353 16037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Refusing update from remote peer 1bd51e6cfd7d4588a8ebdfe215842ac7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 01:03:00.379293 16149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 01:03:00.379631 16118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:00.382236 16150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2130f3384151439b885c797d8eb9cc7f (127.8.254.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:00.382793 16037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2130f3384151439b885c797d8eb9cc7f (127.8.254.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:00.384037 16118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Refusing update from remote peer 1bd51e6cfd7d4588a8ebdfe215842ac7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 01:03:00.385104 16151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:00.385831 16150 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.254.124:40533
I20260924 01:03:00.386379 15982 master.cc:537] Master@127.8.254.124:40533 shutting down...
I20260924 01:03:00.387892 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:00.388274 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 01:03:00.388448 15405 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 01:03:00.404170 15405 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 -> Peer 2130f3384151439b885c797d8eb9cc7f (127.8.254.124:40533): Couldn't send request to peer 2130f3384151439b885c797d8eb9cc7f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 01:03:00.406606 16151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:00.406996 16151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:00.407083 16149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:00.407325 16149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:00.420629 15982 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:00.421100 15982 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:00.421355 15982 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: stopping tablet replica
I20260924 01:03:00.427558 15982 master.cc:552] Master@127.8.254.124:40533 shutdown complete.
I20260924 01:03:00.427675 15982 master_runner.cc:305] Clearing existing system tablet
I20260924 01:03:00.428064 15982 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:00.428198 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:00.430850 15982 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 01:03:00.432415 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:00.432723 15982 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:00.433019 15982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 01:03:00.433132 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
uuid: "2130f3384151439b885c797d8eb9cc7f"
format_stamp: "Formatted at 2026-09-24 01:03:00 on dist-test-slave-5c7l"
server_key: "eb76c03a61394060ad92977a03e5ebfc"
server_key_iv: "6653c35dc08e6589972bb336e1f75e52"
server_key_version: "encryptionkey@0"
I20260924 01:03:00.436744 15982 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:00.441749 15982 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 01:03:00.441936 15982 log.cc:1196] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 01:03:00.442449 15982 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Deleting consensus metadata
I20260924 01:03:00.442714 15982 master_runner.cc:315] Copying system tablet from 127.8.254.126:38457
I20260924 01:03:00.445165 15982 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: tablet copy: Beginning tablet copy session from remote peer at address 127.8.254.126:38457
I20260924 01:03:00.457819 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.445981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.8.254.126:38457)
0924 01:03:00.446280 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:00.446290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:00.446812 (+ 522us) server_negotiation.cc:407] Connection header received
0924 01:03:00.447028 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:00.447034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:00.447125 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:00.447232 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:00.448043 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.448778 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.450009 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.450522 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.450655 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.452466 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:00.452473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:00.452484 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:00.452518 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:00.455237 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.455857 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.455864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.455867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.456006 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:00.456388 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:00.456394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:00.456397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:00.456668 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:00.456850 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:00.457211 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 01:03:00.457428 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 01:03:00.457887 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:00.445973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:38472)
0924 01:03:00.446697 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:00.446713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:00.446888 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:00.447274 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:00.447278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:00.447704 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:00.447887 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.447895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.448931 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.448936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.449843 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:00.449855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:00.450692 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:00.450699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:00.450867 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:00.451519 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:00.451537 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:00.452262 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:00.455372 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.455379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.455383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.455736 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.456121 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:00.456126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:00.456130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:00.456272 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:00.456788 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:00.456795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:00.456970 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 01:03:00.457115 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 01:03:00.457429 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":476,"threads_started":1}
I20260924 01:03:00.458971 15441 tablet_copy_service.cc:140] P 1bd51e6cfd7d4588a8ebdfe215842ac7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2130f3384151439b885c797d8eb9cc7f ({username='oryx', principal='oryx/127.8.254.124@KRBTEST.COM'} at 127.0.0.1:38472)
I20260924 01:03:00.459157 15441 tablet_copy_service.cc:161] P 1bd51e6cfd7d4588a8ebdfe215842ac7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2130f3384151439b885c797d8eb9cc7f at {username='oryx', principal='oryx/127.8.254.124@KRBTEST.COM'} at 127.0.0.1:38472: session id = 2130f3384151439b885c797d8eb9cc7f-00000000000000000000000000000000
I20260924 01:03:00.461789 15441 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 01:03:00.463567 15982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 01:03:00.469079 15982 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: tablet copy: Starting download of 0 data blocks...
I20260924 01:03:00.469399 15982 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: tablet copy: Starting download of 1 WAL segments...
I20260924 01:03:00.472229 15982 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 01:03:00.478490 15441 tablet_copy_service.cc:342] P 1bd51e6cfd7d4588a8ebdfe215842ac7: Request end of tablet copy session 2130f3384151439b885c797d8eb9cc7f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.254.124@KRBTEST.COM'} at 127.0.0.1:38472
I20260924 01:03:00.478654 15441 tablet_copy_service.cc:433] P 1bd51e6cfd7d4588a8ebdfe215842ac7: ending tablet copy session 2130f3384151439b885c797d8eb9cc7f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2130f3384151439b885c797d8eb9cc7f
I20260924 01:03:00.490170 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:00.493285 16189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:00.499331 16190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:00.504160 16192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:00.507249 15982 server_base.cc:1034] running on GCE node
I20260924 01:03:00.507795 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:00.508718 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:00.509924 15982 hybrid_clock.cc:648] HybridClock initialized: now 1790211780509893 us; error 35 us; skew 500 ppm
I20260924 01:03:00.510249 15982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:00 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211780, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:00.513051 15982 init.cc:377] Logged in from keytab as oryx/127.8.254.124@KRBTEST.COM (short username oryx)
I20260924 01:03:00.519507 15982 webserver.cc:460] Webserver started at http://127.8.254.124:42303/ using document root <none> and password file <none>
I20260924 01:03:00.519881 15982 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:00.520028 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:00.522790 15982 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 01:03:00.525312 16199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:00.525835 15982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 01:03:00.526010 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
uuid: "2130f3384151439b885c797d8eb9cc7f"
format_stamp: "Formatted at 2026-09-24 01:03:00 on dist-test-slave-5c7l"
server_key: "eb76c03a61394060ad92977a03e5ebfc"
server_key_iv: "6653c35dc08e6589972bb336e1f75e52"
server_key_version: "encryptionkey@0"
I20260924 01:03:00.526258 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:00.532581 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15482
I20260924 01:03:00.558156 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:40283
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:38457
--webserver_port=34443
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:03:00.563097 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:00.567580 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:00.602955 15982 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.124:40533
I20260924 01:03:00.602980 16251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.124:40533 every 8 connection(s)
I20260924 01:03:00.603828 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
I20260924 01:03:00.611475 16252 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:03:00.615782 16252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Bootstrap starting.
I20260924 01:03:00.633910 16252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:03:00.634480 16252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Bootstrap complete.
I20260924 01:03:00.635390 16252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } }
I20260924 01:03:00.635648 16252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2130f3384151439b885c797d8eb9cc7f, State: Initialized, Role: LEARNER
I20260924 01:03:00.635900 16252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } }
I20260924 01:03:00.638712 16255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:00.639101 16252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:00.639225 16255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: This master's current role is: LEARNER
I20260924 01:03:00.669435 16266 catalog_manager.cc:1259] Loaded cluster ID: aafcc22a693344fc831348d8a2da7bdb
I20260924 01:03:00.669570 16266 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: loading cluster ID for follower catalog manager: success
I20260924 01:03:00.672364 16266 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: acquiring CA information for follower catalog manager: success
I20260924 01:03:00.674181 16266 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 01:03:00.974411 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:00.974711 16204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:00.974821 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:00.982568 16204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:00.982683 16204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:00.982805 16204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:00.994289 16204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:40283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=34443
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:00.995884 16204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:00.997184 16204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:01.017561 16272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:01.017879 16271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:01.021021 16276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:01.021667 16204 server_base.cc:1034] running on GCE node
I20260924 01:03:01.022570 16204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:01.023444 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.004958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.0.0.1:50898)
0924 01:03:01.016211 (+ 11253us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.016233 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.016423 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.017179 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.017187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.017244 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:01.017508 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.017523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.019569 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.019578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.020895 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.020914 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.022590 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.022595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.022942 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.022974 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.023076 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.023192 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11099,"thread_start_us":11000,"threads_started":1}
I20260924 01:03:01.023622 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.005116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.8.254.124:40533)
0924 01:03:01.015991 (+ 10875us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.016000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.016371 (+ 371us) server_negotiation.cc:407] Connection header received
0924 01:03:01.016608 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.016617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.016725 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.016862 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:01.017709 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.019344 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.021094 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.022411 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.022555 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.022798 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.023140 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.023283 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10694,"thread_start_us":10563,"threads_started":1}
I20260924 01:03:01.024062 16204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:01.024892 16227 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260924 01:03:01.025295 16204 hybrid_clock.cc:648] HybridClock initialized: now 1790211781025204 us; error 75 us; skew 500 ppm
I20260924 01:03:01.025523 16255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd51e6cfd7d4588a8ebdfe215842ac7. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:01.025739 16204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:01.025849 16255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: This master's current role is: LEARNER
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:01.030345 16204 init.cc:377] Logged in from keytab as oryx/127.8.254.65@KRBTEST.COM (short username oryx)
I20260924 01:03:01.032691 16204 webserver.cc:460] Webserver started at http://127.8.254.65:34443/ using document root <none> and password file <none>
I20260924 01:03:01.033766 16204 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:01.033993 16204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:01.040704 16204 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 01:03:01.044101 16283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:01.045615 16204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:03:01.045817 16204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "376712ccc3674ae3be2916b277ef8a19"
format_stamp: "Formatted at 2026-09-24 01:02:57 on dist-test-slave-5c7l"
server_key: "a727229019eda9ec072702dd4a93c1d0"
server_key_iv: "593aa4a71fe9f1d9710331d5f6f1dbd3"
server_key_version: "encryptionkey@0"
I20260924 01:03:01.047205 16204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:01.068056 16204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:01.073634 16204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:01.074033 16204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:01.075599 16204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:01.077617 16204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:01.077790 16204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:01.077979 16204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:01.078147 16204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:01.119146 16204 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:40283
I20260924 01:03:01.119251 16395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:40283 every 8 connection(s)
I20260924 01:03:01.120653 16204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:01.127429 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 16204
I20260924 01:03:01.127828 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15616
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.124@KRBTEST.COM
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.66@KRBTEST.COM
I20260924 01:03:01.158882 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.124090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:37853 (local address 127.8.254.124:40533)
0924 01:03:01.124273 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.124279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.126494 (+ 2215us) server_negotiation.cc:407] Connection header received
0924 01:03:01.135416 (+ 8922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.135427 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.135566 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.135655 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.140544 (+ 4889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.142004 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.144242 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.144859 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.144959 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.150094 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.150100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.150103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.150137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.153865 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.155460 (+ 1595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.155467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.155472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.155642 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.156353 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.156359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.156362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.156801 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.156972 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.158236 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.158497 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:03:01.161316 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.123752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:35145 (local address 127.8.254.125:36617)
0924 01:03:01.125069 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.125076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.139482 (+ 14406us) server_negotiation.cc:407] Connection header received
0924 01:03:01.140188 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.140197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.140325 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.140423 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.142091 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.143639 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.145286 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.145976 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.146081 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.151263 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.151272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.151279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.151327 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.157501 (+ 6174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.158698 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.158706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.158709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.158943 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.159536 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.159543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.159549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.159911 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.160062 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.160711 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.160948 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":1008,"threads_started":1}
I20260924 01:03:01.161796 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.125288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:58365)
0924 01:03:01.128922 (+ 3634us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.128946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.129095 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.130584 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.130651 (+ 67us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.131949 (+ 1298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:01.133128 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.133163 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.137217 (+ 4054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.137225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.140275 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.140299 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.147284 (+ 6985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.147291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.147478 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.148143 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:01.148165 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:01.152489 (+ 4324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:01.157814 (+ 5325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.157819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.157823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.158579 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.159033 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.159038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.159041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.159246 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.159944 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:01.159952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:01.160098 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.160246 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.160537 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"spinlock_wait_cycles":36096,"thread_start_us":538,"threads_started":1}
I20260924 01:03:01.162322 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.125359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.8.254.65:35145)
0924 01:03:01.139328 (+ 13969us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.139356 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.139993 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.140718 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.140726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.141468 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:01.141874 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.141889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.143893 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.143906 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.145119 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.145136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.146412 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.146420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.146641 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.147613 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:01.147644 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:01.150981 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:01.157682 (+ 6701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.157687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.157691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.158534 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.159113 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.159121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.159126 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.159375 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.160179 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:01.160187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:01.160374 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.161520 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.161824 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13767,"thread_start_us":14162,"threads_started":1}
I20260924 01:03:01.163378 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.123901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:58365 (local address 127.8.254.126:38457)
0924 01:03:01.124674 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.124680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.129424 (+ 4744us) server_negotiation.cc:407] Connection header received
0924 01:03:01.129534 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.129542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.129708 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.129839 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.133477 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.135133 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.146095 (+ 10962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.146824 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.146936 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.152765 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.152772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.152777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.152827 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.157650 (+ 4823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.158731 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.158737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.158741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.158892 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.159375 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.159380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.159382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.159741 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.159872 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.162692 (+ 2820us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.162985 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":503,"threads_started":1}
I20260924 01:03:01.163906 16398 heartbeater.cc:347] Connected to a master server at 127.8.254.125:36617
I20260924 01:03:01.164266 16397 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38457
I20260924 01:03:01.161840 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.124652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.8.254.65:37853)
0924 01:03:01.126296 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.126342 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.128588 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.135935 (+ 7347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.135944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.136621 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:01.136989 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.137004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.142259 (+ 5255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.142268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.144055 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.144069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.145318 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.145325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.145659 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.146445 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:01.146481 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:01.149844 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:01.154470 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.154481 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.154501 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.155263 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.155810 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.155817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.155820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.156191 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.157162 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:01.157172 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:01.157740 (+ 568us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.159461 (+ 1721us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.160056 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"thread_start_us":982,"threads_started":1}
I20260924 01:03:01.164426 16397 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:01.165447 16397 heartbeater.cc:511] Master 127.8.254.126:38457 requested a full tablet report, sending...
I20260924 01:03:01.165478 16396 heartbeater.cc:347] Connected to a master server at 127.8.254.124:40533
I20260924 01:03:01.165616 16396 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:01.166126 16396 heartbeater.cc:511] Master 127.8.254.124:40533 requested a full tablet report, sending...
I20260924 01:03:01.164299 16398 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:01.166445 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.129035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.8.254.66:41739)
0924 01:03:01.144569 (+ 15534us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.144579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.144610 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:01.144676 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.144687 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.145000 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.145174 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.147399 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.149019 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.151078 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.151700 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.151859 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.157927 (+ 6068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.157946 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.157964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.158017 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.162107 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.163407 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.163415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.163419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.163591 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.164182 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.164189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.164193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.164739 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.165417 (+ 678us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.165615 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.165926 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15348,"thread_start_us":15171,"threads_started":1}
I20260924 01:03:01.166735 16398 heartbeater.cc:511] Master 127.8.254.125:36617 requested a full tablet report, sending...
I20260924 01:03:01.167965 15421 ts_manager.cc:194] Re-registered known tserver with Master: 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283)
I20260924 01:03:01.168355 16217 ts_manager.cc:194] Registered new tserver with Master: 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283)
I20260924 01:03:01.168594 15739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:01.169256 15421 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:58365
I20260924 01:03:01.170390 16027 ts_manager.cc:194] Registered new tserver with Master: 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283)
I20260924 01:03:01.261760 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15616
I20260924 01:03:01.279521 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:41739
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:38457
--webserver_port=37341
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:03:01.504694 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: NON_VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: true } } }
I20260924 01:03:01.505167 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:01.506981 16179 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: attempting to promote NON_VOTER 2130f3384151439b885c797d8eb9cc7f to VOTER
I20260924 01:03:01.507575 16179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:01.515654 16037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Refusing update from remote peer 1bd51e6cfd7d4588a8ebdfe215842ac7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 01:03:01.515709 16227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 LEARNER]: Refusing update from remote peer 1bd51e6cfd7d4588a8ebdfe215842ac7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 01:03:01.516310 16179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 01:03:01.516697 16410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 01:03:01.519789 16179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2130f3384151439b885c797d8eb9cc7f (127.8.254.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.522559 16037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2130f3384151439b885c797d8eb9cc7f (127.8.254.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.522979 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.523267 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:01.523550 16409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.523921 16409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:01.527209 16415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.527592 16415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:03:01.529346 16410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.529626 16410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:01.530155 16227 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2130f3384151439b885c797d8eb9cc7f (127.8.254.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.533574 16416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:01.533907 16416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: This master's current role is: FOLLOWER
W20260924 01:03:01.625691 16408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:01.625999 16408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:01.626271 16408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:01.634742 16408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:01.634888 16408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:01.635066 16408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:01.645618 16408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:41739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=37341
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:01.647759 16408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:01.649843 16408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:01.664060 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:01.664143 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:01.664153 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:01.667886 16408 server_base.cc:1034] running on GCE node
I20260924 01:03:01.668601 16408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:01.669708 16408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:01.670925 16408 hybrid_clock.cc:648] HybridClock initialized: now 1790211781670863 us; error 49 us; skew 500 ppm
I20260924 01:03:01.671283 16408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:01.674642 16408 init.cc:377] Logged in from keytab as oryx/127.8.254.66@KRBTEST.COM (short username oryx)
I20260924 01:03:01.676283 16408 webserver.cc:460] Webserver started at http://127.8.254.66:37341/ using document root <none> and password file <none>
I20260924 01:03:01.677068 16408 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:01.677239 16408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:01.683226 16408 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 01:03:01.686273 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:01.687755 16408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:03:01.687965 16408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "a2b7ef6ee4f341db9e79fb6952e03a83"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "2f096966a7caa0b8afeb59af22f462fb"
server_key_iv: "201493323a0d45ce5edbae385ffec62e"
server_key_version: "encryptionkey@0"
I20260924 01:03:01.689010 16408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:01.705940 16408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:01.709916 16408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:01.710242 16408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:01.711315 16408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:01.712769 16408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:01.712900 16408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:01.713038 16408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:01.713150 16408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:01.747349 16408 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:41739
I20260924 01:03:01.747423 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:41739 every 8 connection(s)
I20260924 01:03:01.748873 16408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:01.752122 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 16408
I20260924 01:03:01.752403 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15750
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.124@KRBTEST.COM
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211781, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
Sep 24 01:03:01 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.67@KRBTEST.COM
I20260924 01:03:01.795326 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.755853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:44119 (local address 127.8.254.126:38457)
0924 01:03:01.766693 (+ 10840us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.766702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.766731 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:03:01.766820 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.766829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.766943 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.767106 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.770521 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.771644 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.775092 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.775726 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.775830 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.785732 (+ 9902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.785743 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.785750 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.785806 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.790060 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.791286 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.791292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.791296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.791476 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.792090 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.792095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.792098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.792788 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.792994 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.794326 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.794720 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10644,"thread_start_us":10531,"threads_started":1}
I20260924 01:03:01.796542 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.751894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:42181 (local address 127.8.254.124:40533)
0924 01:03:01.768532 (+ 16638us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.768540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.768561 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:03:01.768629 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.768635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.768718 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.769764 (+ 1046us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.770485 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.771672 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.773847 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.774525 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.774643 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.786365 (+ 11722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.786375 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.786383 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.786435 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.790046 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.791286 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.791292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.791296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.791476 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.792087 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.792093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.792096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.792520 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.792669 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.795856 (+ 3187us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.796145 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16516,"thread_start_us":16385,"threads_started":1}
I20260924 01:03:01.796859 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.755962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.8.254.66:38133)
0924 01:03:01.758578 (+ 2616us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.758605 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.762940 (+ 4335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.768015 (+ 5075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.768020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.769125 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:01.770607 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.770621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.772221 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.772229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.774162 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.774178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.775252 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.775259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.775479 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.782064 (+ 6585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:01.782146 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:01.787170 (+ 5024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:01.792608 (+ 5438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.792615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.792619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.793410 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.793998 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.794003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.794007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.794272 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.795212 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:01.795217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:01.795429 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.795606 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.795932 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2494,"mutex_wait_us":685,"spinlock_wait_cycles":54912,"thread_start_us":2395,"threads_started":1}
I20260924 01:03:01.797575 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.755432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.8.254.66:42181)
0924 01:03:01.756420 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.756491 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.763391 (+ 6900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.768905 (+ 5514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.768910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.769355 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:01.770234 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.770267 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.771866 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.771872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.773566 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.773696 (+ 130us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.774974 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.774984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.775385 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.782120 (+ 6735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:01.782164 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:01.786039 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:01.790262 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.790272 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.790294 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.791126 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.791616 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.791621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.791624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.791940 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.792776 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:01.792785 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:01.793393 (+ 608us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.793728 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.794352 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":69,"spinlock_wait_cycles":7424,"thread_start_us":1668,"threads_started":1}
I20260924 01:03:01.797618 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.754039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:38133 (local address 127.8.254.125:36617)
0924 01:03:01.766721 (+ 12682us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.766729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.766756 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:03:01.766891 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.766899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.767016 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.769033 (+ 2017us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.770830 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.772024 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.774364 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.775023 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.775866 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.787492 (+ 11626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.787501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.787508 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.787561 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.792413 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.793658 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.793664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.793667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.793870 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.794428 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.794433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.794436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.795061 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.796757 (+ 1696us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.796966 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.797206 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12516,"thread_start_us":12376,"threads_started":1}
I20260924 01:03:01.797712 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.752685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:44119)
0924 01:03:01.759961 (+ 7276us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:01.759985 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:01.763483 (+ 3498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:01.767760 (+ 4277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:01.767779 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:01.769725 (+ 1946us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:01.770273 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.770293 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.771841 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.771846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.774918 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:01.774932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.776109 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.776113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:01.776366 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.782120 (+ 5754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:01.782154 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:01.785407 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:01.790268 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.790272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.790294 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.791126 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.791616 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:01.791621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:01.791625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:01.791940 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:01.793074 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:01.793080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:01.793450 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 01:03:01.793837 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 01:03:01.795821 (+ 1984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1414,"spinlock_wait_cycles":88064,"thread_start_us":7619,"threads_started":1}
I20260924 01:03:01.798765 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:01.753889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.8.254.67:32783)
0924 01:03:01.764290 (+ 10401us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:01.764303 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:01.767738 (+ 3435us) server_negotiation.cc:407] Connection header received
0924 01:03:01.767949 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:01.767962 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:01.768260 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:01.768441 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:01.770813 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.772391 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.774255 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:01.775235 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:01.775356 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:01.789992 (+ 14636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:01.790026 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:01.790048 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:01.790106 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:01.794822 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.795911 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.795920 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.795923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.796092 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:01.796598 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:01.796607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:01.796612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:01.797059 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:01.797368 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:01.798045 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 01:03:01.798370 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10162,"thread_start_us":10420,"threads_started":1}
I20260924 01:03:01.798986 16546 heartbeater.cc:347] Connected to a master server at 127.8.254.125:36617
I20260924 01:03:01.799152 16544 heartbeater.cc:347] Connected to a master server at 127.8.254.124:40533
I20260924 01:03:01.799374 16546 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:01.799378 16544 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:01.799372 16545 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38457
I20260924 01:03:01.800305 16545 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:01.800537 16544 heartbeater.cc:511] Master 127.8.254.124:40533 requested a full tablet report, sending...
I20260924 01:03:01.800606 16546 heartbeater.cc:511] Master 127.8.254.125:36617 requested a full tablet report, sending...
I20260924 01:03:01.800928 16545 heartbeater.cc:511] Master 127.8.254.126:38457 requested a full tablet report, sending...
I20260924 01:03:01.801103 15873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:01.802624 16027 ts_manager.cc:194] Registered new tserver with Master: a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66:41739)
I20260924 01:03:01.802659 15421 ts_manager.cc:194] Re-registered known tserver with Master: a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66:41739)
I20260924 01:03:01.804164 15421 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:44119
I20260924 01:03:01.805279 16217 ts_manager.cc:194] Registered new tserver with Master: a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66:41739)
I20260924 01:03:01.890650 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15750
I20260924 01:03:01.910432 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:32783
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:38457
--webserver_port=42839
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:03:02.172906 16397 heartbeater.cc:503] Master 127.8.254.126:38457 was elected leader, sending a full tablet report...
W20260924 01:03:02.265843 16557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:02.266147 16557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:02.266336 16557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:02.273872 16557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:02.273993 16557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:02.274111 16557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:02.284446 16557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:32783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=42839
--tserver_master_addrs=127.8.254.126:38457,127.8.254.125:36617,127.8.254.124:40533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:02.285792 16557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:02.287189 16557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:02.301774 16562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:02.301851 16563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:02.301873 16565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:02.302778 16557 server_base.cc:1034] running on GCE node
I20260924 01:03:02.303426 16557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:02.304491 16557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:02.305758 16557 hybrid_clock.cc:648] HybridClock initialized: now 1790211782305699 us; error 46 us; skew 500 ppm
I20260924 01:03:02.306126 16557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:02 dist-test-slave-5c7l krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211782, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:02.316258 16557 init.cc:377] Logged in from keytab as oryx/127.8.254.67@KRBTEST.COM (short username oryx)
I20260924 01:03:02.317934 16557 webserver.cc:460] Webserver started at http://127.8.254.67:42839/ using document root <none> and password file <none>
I20260924 01:03:02.318826 16557 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:02.319006 16557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:02.324894 16557 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 01:03:02.328016 16572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:02.329610 16557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:03:02.329807 16557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "3fef41032fb241dd8d8c1bcdf5d09396"
format_stamp: "Formatted at 2026-09-24 01:02:58 on dist-test-slave-5c7l"
server_key: "d4877aa394868b131b0eb2ab9bd3bf02"
server_key_iv: "b68dd1ccfa164e9315db497d1d1fd581"
server_key_version: "encryptionkey@0"
I20260924 01:03:02.330888 16557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:02.377430 16557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:02.382007 16557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:02.382380 16557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:02.383391 16557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:02.384745 16557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:02.384868 16557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:02.385000 16557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:02.385089 16557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:02.419039 16557 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:32783
I20260924 01:03:02.419130 16684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:32783 every 8 connection(s)
I20260924 01:03:02.420528 16557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:02.426963 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 16557
Sep 24 01:03:02 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211782, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.125@KRBTEST.COM
Sep 24 01:03:02 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211782, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.124@KRBTEST.COM
Sep 24 01:03:02 dist-test-slave-5c7l krb5kdc[15376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211782, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:02.464038 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.424456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:58429)
0924 01:03:02.430903 (+ 6447us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.430954 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.434103 (+ 3149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.440352 (+ 6249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.440365 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.440954 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:02.441326 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.441342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.443461 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.443469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.446757 (+ 3288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.446774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.447877 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.447887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.448106 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.448738 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:02.448762 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:02.451434 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:02.456789 (+ 5355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:02.456806 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:02.456837 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:02.457698 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:02.458355 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:02.458363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:02.458369 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:02.458754 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:02.460193 (+ 1439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:02.460207 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:02.460870 (+ 663us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.461261 (+ 391us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.462078 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6107,"thread_start_us":10942,"threads_started":1}
I20260924 01:03:02.466811 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.423634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:58429 (local address 127.8.254.126:38457)
0924 01:03:02.439895 (+ 16261us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.439903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.439924 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:03:02.440005 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.440013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.440108 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.441531 (+ 1423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:02.441748 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.443256 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.446937 (+ 3681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.447637 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.448452 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.451754 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:02.451761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:02.451768 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:02.451814 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:02.456541 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.457929 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.457937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.457944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.458142 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.459448 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.459462 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.459468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.459991 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:02.465559 (+ 5568us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.465793 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.466156 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16074,"thread_start_us":15974,"threads_started":1}
I20260924 01:03:02.471187 16686 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38457
I20260924 01:03:02.471704 16686 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:02.472913 16686 heartbeater.cc:511] Master 127.8.254.126:38457 requested a full tablet report, sending...
I20260924 01:03:02.475668 15421 ts_manager.cc:194] Re-registered known tserver with Master: 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:02.476907 15421 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:58429
I20260924 01:03:02.477653 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.425559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.8.254.67:58773)
0924 01:03:02.435746 (+ 10187us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.435763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.435868 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.436719 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.436727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.437490 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:02.438137 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.438155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.439762 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.439768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.442217 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.442234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.443376 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.443385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.443682 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.444484 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:02.444522 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:02.448324 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:02.470821 (+ 22497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:02.470826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:02.470831 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:02.474477 (+ 3646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:02.475570 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:02.475576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:02.475579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:02.475767 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:02.476534 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:02.476539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:02.476713 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.476853 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.477163 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10061,"spinlock_wait_cycles":32128,"thread_start_us":4912,"threads_started":1}
I20260924 01:03:02.479126 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.425077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.8.254.67:46765)
0924 01:03:02.434746 (+ 9669us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.434771 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.434943 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.439359 (+ 4416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.439364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.439919 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:02.440286 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.440298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.441876 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.441884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.443420 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.443434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.444447 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.444453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.444649 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.445227 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:02.445249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:02.448589 (+ 3340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:02.476149 (+ 27560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:02.476154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:02.476157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:02.476657 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:02.477208 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:02.477212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:02.477215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:02.477380 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:02.478057 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:02.478065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:02.478281 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.478427 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.478724 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9518,"thread_start_us":9355,"threads_started":1}
I20260924 01:03:02.480108 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.423980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:46765 (local address 127.8.254.124:40533)
0924 01:03:02.438969 (+ 14989us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.438977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.439021 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:03:02.439092 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.439100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.439193 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.439321 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:02.440514 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.441677 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.443605 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.444217 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.444962 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.449807 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:02.449816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:02.449823 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:02.449885 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:02.475996 (+ 26111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.476875 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.476881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.476884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.477064 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.477523 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.477527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.477530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.477898 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:02.479364 (+ 1466us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.479506 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.479749 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14857,"thread_start_us":14736,"threads_started":1}
I20260924 01:03:02.480911 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.423633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:58773 (local address 127.8.254.125:36617)
0924 01:03:02.436129 (+ 12496us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.436138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.436164 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:03:02.436242 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.436249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.436352 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.436496 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:02.438378 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.439562 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.442461 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.443105 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.444128 (+ 1023us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.448875 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:02.448882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:02.448894 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:02.448954 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:02.470659 (+ 21705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.475221 (+ 4562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.475227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.475232 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.475403 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.475945 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.475951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.475954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.476369 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:02.479907 (+ 3538us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.480052 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.480454 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12338,"thread_start_us":12982,"threads_started":1}
I20260924 01:03:02.481070 16685 heartbeater.cc:347] Connected to a master server at 127.8.254.124:40533
I20260924 01:03:02.481237 16685 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:02.481678 16685 heartbeater.cc:511] Master 127.8.254.124:40533 requested a full tablet report, sending...
I20260924 01:03:02.482543 16687 heartbeater.cc:347] Connected to a master server at 127.8.254.125:36617
I20260924 01:03:02.482676 16687 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:02.483040 16687 heartbeater.cc:511] Master 127.8.254.125:36617 requested a full tablet report, sending...
I20260924 01:03:02.483933 16217 ts_manager.cc:194] Registered new tserver with Master: 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:02.485673 16027 ts_manager.cc:194] Registered new tserver with Master: 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:02.493722 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.449579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.8.254.124:40533)
0924 01:03:02.450543 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.450549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.468659 (+ 18110us) server_negotiation.cc:407] Connection header received
0924 01:03:02.468977 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.468983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.469074 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.469160 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:02.472386 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.474045 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.476272 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.476962 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.477063 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.484259 (+ 7196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:02.484267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:02.484271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:02.484314 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:02.489167 (+ 4853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.490083 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.490092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.490098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.490312 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.490865 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.490871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.490876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.491264 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:02.491513 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.492987 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.493268 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":669,"threads_started":1}
I20260924 01:03:02.495958 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.448915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.8.254.125:36617)
0924 01:03:02.477688 (+ 28773us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.477694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.477713 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:03:02.477776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.477782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.477868 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.479471 (+ 1603us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:02.481182 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.482324 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.484482 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.485202 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.485924 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.488627 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:02.488636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:02.488644 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:02.488685 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:02.491667 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.492828 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.492832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.492835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.493015 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.493518 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.493524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.493528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.494200 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:02.494539 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.495337 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.495570 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28634,"thread_start_us":29035,"threads_started":1}
I20260924 01:03:02.497944 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.451825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.8.254.126:38457)
0924 01:03:02.467645 (+ 15820us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.467654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.478189 (+ 10535us) server_negotiation.cc:407] Connection header received
0924 01:03:02.478405 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.478412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.478502 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.478602 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:02.483548 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.484616 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.486808 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.487519 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.487653 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.491323 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:02.491332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:02.491337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:02.491377 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:02.494962 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.495959 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.495965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.495969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.496143 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:02.496572 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:02.496577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:02.496580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:02.496939 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:02.497191 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.497544 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.497720 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15658,"thread_start_us":16042,"threads_started":1}
I20260924 01:03:02.502027 15421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:02.508868 15421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:02.545811 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.526182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40283 (local address 127.0.0.1:53558)
0924 01:03:02.537301 (+ 11119us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.537328 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.537794 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.539032 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.539039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.539078 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:02.539598 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.539614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.541422 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.541428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.543650 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.543665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.544971 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.544977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.545146 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.545181 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.545306 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.545453 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10929,"thread_start_us":12182,"threads_started":1}
I20260924 01:03:02.546393 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.526447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.8.254.65:40283)
0924 01:03:02.536793 (+ 10346us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.536807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.537485 (+ 678us) server_negotiation.cc:407] Connection header received
0924 01:03:02.538039 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.538054 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.538466 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.538703 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:02.539803 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.541271 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.543820 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.544758 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.544950 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.545176 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.545350 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.545515 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10074,"thread_start_us":10364,"threads_started":1}
I20260924 01:03:02.547521 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.527293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.8.254.66:41739)
0924 01:03:02.537443 (+ 10150us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.537455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.539736 (+ 2281us) server_negotiation.cc:407] Connection header received
0924 01:03:02.539838 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.539845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.540029 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.540224 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:02.540922 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.542301 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.544839 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.546307 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.546454 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.546669 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.547200 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.547332 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9971,"thread_start_us":9870,"threads_started":1}
I20260924 01:03:02.547631 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.527006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41739 (local address 127.0.0.1:53086)
0924 01:03:02.539462 (+ 12456us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.539482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.539596 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.540244 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.540249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.540274 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:02.540710 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.540726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.542498 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.542505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.544639 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.544653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.546826 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.546832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.547004 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.547037 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.547130 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.547257 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12291,"mutex_wait_us":1,"thread_start_us":10519,"threads_started":1}
I20260924 01:03:02.550189 16331 tablet_service.cc:1467] Processing CreateTablet for tablet d3d8fd58a5264d0b99ea9af44688fd1f (DEFAULT_TABLE table=test-table [id=55843c40a51247ef9f732d1cb591d7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:02.550784 16479 tablet_service.cc:1467] Processing CreateTablet for tablet d3d8fd58a5264d0b99ea9af44688fd1f (DEFAULT_TABLE table=test-table [id=55843c40a51247ef9f732d1cb591d7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:02.551823 16331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d8fd58a5264d0b99ea9af44688fd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:02.551980 16479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d8fd58a5264d0b99ea9af44688fd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:02.553508 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.526509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:32783 (local address 127.0.0.1:39006)
0924 01:03:02.538639 (+ 12130us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.538664 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.538814 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.540456 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.540461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.540482 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:02.540830 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.540865 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.543808 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.543812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.545906 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.545921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.552553 (+ 6632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.552560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.552749 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.552784 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.552907 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.553134 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12017,"thread_start_us":12585,"threads_started":1}
I20260924 01:03:02.553516 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.527289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.8.254.67:32783)
0924 01:03:02.538119 (+ 10830us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.538130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.538769 (+ 639us) server_negotiation.cc:407] Connection header received
0924 01:03:02.538956 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.538964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.539345 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.539560 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:02.541033 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.542908 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.550647 (+ 7739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.552244 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.552402 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.552615 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.552974 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.553138 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10593,"thread_start_us":10367,"threads_started":1}
I20260924 01:03:02.557617 16620 tablet_service.cc:1467] Processing CreateTablet for tablet d3d8fd58a5264d0b99ea9af44688fd1f (DEFAULT_TABLE table=test-table [id=55843c40a51247ef9f732d1cb591d7fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:02.559137 16620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d8fd58a5264d0b99ea9af44688fd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:02.564567 16719 tablet_bootstrap.cc:492] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Bootstrap starting.
I20260924 01:03:02.568264 16719 tablet_bootstrap.cc:654] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:02.569512 16719 log.cc:824] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:02.569808 16720 tablet_bootstrap.cc:492] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Bootstrap starting.
I20260924 01:03:02.574682 16720 tablet_bootstrap.cc:654] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:02.576301 16720 log.cc:824] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:02.579864 16719 tablet_bootstrap.cc:492] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: No bootstrap required, opened a new log
I20260924 01:03:02.580024 16720 tablet_bootstrap.cc:492] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: No bootstrap required, opened a new log
I20260924 01:03:02.580070 16722 tablet_bootstrap.cc:492] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Bootstrap starting.
I20260924 01:03:02.580230 16719 ts_tablet_manager.cc:1397] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260924 01:03:02.580300 16720 ts_tablet_manager.cc:1397] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 01:03:02.583801 16722 tablet_bootstrap.cc:654] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:02.585037 16722 log.cc:824] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:02.588490 16722 tablet_bootstrap.cc:492] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: No bootstrap required, opened a new log
I20260924 01:03:02.588783 16722 ts_tablet_manager.cc:1397] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260924 01:03:02.589315 16719 raft_consensus.cc:348] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.589699 16719 raft_consensus.cc:374] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:02.589838 16719 raft_consensus.cc:729] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b7ef6ee4f341db9e79fb6952e03a83, State: Initialized, Role: FOLLOWER
I20260924 01:03:02.590499 16719 consensus_queue.cc:260] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.590653 16720 raft_consensus.cc:348] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.591080 16720 raft_consensus.cc:374] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:02.591239 16720 raft_consensus.cc:729] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376712ccc3674ae3be2916b277ef8a19, State: Initialized, Role: FOLLOWER
I20260924 01:03:02.591835 16545 heartbeater.cc:503] Master 127.8.254.126:38457 was elected leader, sending a full tablet report...
I20260924 01:03:02.592134 16720 consensus_queue.cc:260] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.597995 16719 ts_tablet_manager.cc:1428] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20260924 01:03:02.600960 16720 ts_tablet_manager.cc:1428] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20260924 01:03:02.604835 16722 raft_consensus.cc:348] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.606774 16722 raft_consensus.cc:374] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:02.606950 16722 raft_consensus.cc:729] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fef41032fb241dd8d8c1bcdf5d09396, State: Initialized, Role: FOLLOWER
I20260924 01:03:02.607821 16722 consensus_queue.cc:260] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.610076 16722 ts_tablet_manager.cc:1428] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Time spent starting tablet: real 0.021s user 0.015s sys 0.000s
I20260924 01:03:02.610307 16686 heartbeater.cc:503] Master 127.8.254.126:38457 was elected leader, sending a full tablet report...
W20260924 01:03:02.624661 16399 tablet.cc:2369] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:03:02.679201 16688 tablet.cc:2369] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:03:02.761940 16547 tablet.cc:2369] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:02.889051 16725 raft_consensus.cc:482] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:02.889437 16725 raft_consensus.cc:504] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.891433 16725 leader_election.cc:288] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283), 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:02.899582 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.891930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:40159 (local address 127.8.254.67:32783)
0924 01:03:02.892127 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.892134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.892916 (+ 782us) server_negotiation.cc:407] Connection header received
0924 01:03:02.893178 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.893181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.893282 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.893377 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:02.894290 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.895447 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.897569 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.898595 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.898720 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.898774 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.899196 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.899337 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:03:02.899657 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.891921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:32783 (local address 127.8.254.66:40159)
0924 01:03:02.892766 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.892784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.892968 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.893436 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.893446 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.893482 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:02.893975 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.893987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.895663 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.895667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.897391 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.897405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.898799 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.898809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.899014 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.899054 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.899148 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.899307 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":51,"spinlock_wait_cycles":6912,"thread_start_us":462,"threads_started":1}
I20260924 01:03:02.900409 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.891937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:50549 (local address 127.8.254.65:40283)
0924 01:03:02.892131 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:02.892137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:02.892916 (+ 779us) server_negotiation.cc:407] Connection header received
0924 01:03:02.893125 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:02.893129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:02.893194 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:02.893287 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:02.894126 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.895096 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.897865 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.899010 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.899181 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.899247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:02.899702 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 01:03:02.899842 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:03:02.900506 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:02.891776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40283 (local address 127.8.254.66:50549)
0924 01:03:02.892764 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:02.892790 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:02.892968 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:02.893336 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:02.893353 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:02.893384 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:02.893933 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.893952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.895251 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.895258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.897695 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:02.897712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:02.899276 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:02.899283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:02.899512 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:02.899543 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 01:03:02.900000 (+ 457us) client_negotiation.cc:240] Negotiation successful
0924 01:03:02.900121 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":1,"spinlock_wait_cycles":28928,"thread_start_us":556,"threads_started":1}
I20260924 01:03:02.900888 16640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d8fd58a5264d0b99ea9af44688fd1f" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" is_pre_election: true
I20260924 01:03:02.901432 16640 raft_consensus.cc:2454] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 0.
I20260924 01:03:02.901438 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d8fd58a5264d0b99ea9af44688fd1f" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376712ccc3674ae3be2916b277ef8a19" is_pre_election: true
I20260924 01:03:02.901985 16351 raft_consensus.cc:2454] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 0.
I20260924 01:03:02.902081 16435 leader_election.cc:302] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fef41032fb241dd8d8c1bcdf5d09396, a2b7ef6ee4f341db9e79fb6952e03a83; no voters:
I20260924 01:03:02.902540 16725 raft_consensus.cc:2781] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:02.902684 16725 raft_consensus.cc:482] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:02.902808 16725 raft_consensus.cc:3037] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:02.905026 16725 raft_consensus.cc:504] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.905767 16725 leader_election.cc:288] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 election: Requested vote from peers 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283), 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:02.906172 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d8fd58a5264d0b99ea9af44688fd1f" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376712ccc3674ae3be2916b277ef8a19"
I20260924 01:03:02.906296 16640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d8fd58a5264d0b99ea9af44688fd1f" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fef41032fb241dd8d8c1bcdf5d09396"
I20260924 01:03:02.906457 16351 raft_consensus.cc:3037] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:02.906589 16640 raft_consensus.cc:3037] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:02.908716 16351 raft_consensus.cc:2454] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 1.
I20260924 01:03:02.908865 16640 raft_consensus.cc:2454] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 1.
I20260924 01:03:02.909204 16434 leader_election.cc:302] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376712ccc3674ae3be2916b277ef8a19, a2b7ef6ee4f341db9e79fb6952e03a83; no voters:
I20260924 01:03:02.909526 16725 raft_consensus.cc:2781] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:02.910053 16725 raft_consensus.cc:686] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 LEADER]: Becoming Leader. State: Replica: a2b7ef6ee4f341db9e79fb6952e03a83, State: Running, Role: LEADER
I20260924 01:03:02.910650 16725 consensus_queue.cc:237] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } }
I20260924 01:03:02.915910 15421 catalog_manager.cc:5495] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } health_report { overall_health: HEALTHY } } }
I20260924 01:03:03.012198 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:03.006315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.8.254.66:41739)
0924 01:03:03.006505 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:03.006510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:03.006754 (+ 244us) server_negotiation.cc:407] Connection header received
0924 01:03:03.006995 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:03.007000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:03.007105 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:03.007331 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:03.007738 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.008514 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.009835 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.010320 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.010424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:03.011022 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:03.011548 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:03.011659 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:03.011785 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 01:03:03.011859 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:03:03.020743 16640 raft_consensus.cc:1261] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Refusing update from remote peer a2b7ef6ee4f341db9e79fb6952e03a83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:03.020740 16351 raft_consensus.cc:1261] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Refusing update from remote peer a2b7ef6ee4f341db9e79fb6952e03a83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:03.021781 16730 consensus_queue.cc:1035] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:03.022148 16725 consensus_queue.cc:1035] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:03.033651 16734 mvcc.cc:205] Tried to move back new op lower bound from 7332707463243055104 to 7332707462806315008. Current Snapshot: MvccSnapshot[applied={T|T < 7332707463243055104}]
I20260924 01:03:03.040314 16736 mvcc.cc:205] Tried to move back new op lower bound from 7332707463243055104 to 7332707462806315008. Current Snapshot: MvccSnapshot[applied={T|T < 7332707463243055104}]
I20260924 01:03:03.052479 15420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:03.052793 15420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:03.060181 15420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bd51e6cfd7d4588a8ebdfe215842ac7: Sending DeleteTablet for 3 replicas of tablet d3d8fd58a5264d0b99ea9af44688fd1f
I20260924 01:03:03.061276 16331 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d8fd58a5264d0b99ea9af44688fd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:03 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:53558
I20260924 01:03:03.061852 16620 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d8fd58a5264d0b99ea9af44688fd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:03 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:39006
I20260924 01:03:03.061887 16479 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d8fd58a5264d0b99ea9af44688fd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:03 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:53086
I20260924 01:03:03.077611 16743 tablet_replica.cc:330] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: stopping tablet replica
I20260924 01:03:03.077777 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15389
I20260924 01:03:03.078132 16743 raft_consensus.cc:2229] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:03.078998 15451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:03.084971 16743 raft_consensus.cc:2258] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:03.088256 16743 ts_tablet_manager.cc:1905] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:03.092438 16744 tablet_replica.cc:330] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: stopping tablet replica
I20260924 01:03:03.092922 16744 raft_consensus.cc:2229] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:03.093621 16744 raft_consensus.cc:2258] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:03.094753 16745 tablet_replica.cc:330] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: stopping tablet replica
I20260924 01:03:03.097604 16745 raft_consensus.cc:2229] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:03.098169 16745 raft_consensus.cc:2258] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:03.099460 16744 ts_tablet_manager.cc:1905] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:03.103497 16745 ts_tablet_manager.cc:1905] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:03.110850 16745 ts_tablet_manager.cc:1918] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:03.111044 16745 log.cc:1196] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/d3d8fd58a5264d0b99ea9af44688fd1f
I20260924 01:03:03.111624 16745 ts_tablet_manager.cc:1939] T d3d8fd58a5264d0b99ea9af44688fd1f P 3fef41032fb241dd8d8c1bcdf5d09396: Deleting consensus metadata
I20260924 01:03:03.113011 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:03.065786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.8.254.125:36617)
0924 01:03:03.073402 (+ 7616us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:03.073409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:03.076674 (+ 3265us) server_negotiation.cc:407] Connection header received
0924 01:03:03.076772 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:03.076777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:03.076866 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:03.076974 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:03.091095 (+ 14121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.092388 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.094863 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.095585 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.095696 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:03.105292 (+ 9596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:03.105300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:03.105306 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:03.105358 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:03.109145 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.110133 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.110139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.110145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.110333 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.110834 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.110840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.110843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.111549 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:03.111796 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:03.112381 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 01:03:03.112625 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7457,"thread_start_us":7290,"threads_started":1}
I20260924 01:03:03.116734 16743 ts_tablet_manager.cc:1918] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:03.116938 16743 log.cc:1196] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/d3d8fd58a5264d0b99ea9af44688fd1f
I20260924 01:03:03.117506 16743 ts_tablet_manager.cc:1939] T d3d8fd58a5264d0b99ea9af44688fd1f P 376712ccc3674ae3be2916b277ef8a19: Deleting consensus metadata
I20260924 01:03:03.132489 16744 ts_tablet_manager.cc:1918] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:03.132710 16744 log.cc:1196] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/d3d8fd58a5264d0b99ea9af44688fd1f
I20260924 01:03:03.133399 16744 ts_tablet_manager.cc:1939] T d3d8fd58a5264d0b99ea9af44688fd1f P a2b7ef6ee4f341db9e79fb6952e03a83: Deleting consensus metadata
I20260924 01:03:03.143183 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:03.076809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.8.254.124:40533)
0924 01:03:03.091799 (+ 14990us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:03.091805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:03.115568 (+ 23763us) server_negotiation.cc:407] Connection header received
0924 01:03:03.115735 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:03.115741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:03.115835 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:03.115939 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:03.122401 (+ 6462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.123602 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.125539 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.126274 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.126388 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:03.134406 (+ 8018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:03.134414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:03.134420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:03.134469 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:03.138126 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.139831 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.139839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.139843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.140021 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.140496 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.140501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.140505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.140899 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:03.141128 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:03.142535 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0924 01:03:03.142803 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14830,"thread_start_us":14992,"threads_started":1}
I20260924 01:03:03.228992 15405 catalog_manager.cc:4841] TS a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66:41739): tablet d3d8fd58a5264d0b99ea9af44688fd1f (table test-table [id=55843c40a51247ef9f732d1cb591d7fc]) successfully deleted
I20260924 01:03:03.229188 15408 catalog_manager.cc:4841] TS 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783): tablet d3d8fd58a5264d0b99ea9af44688fd1f (table test-table [id=55843c40a51247ef9f732d1cb591d7fc]) successfully deleted
I20260924 01:03:03.228953 15407 catalog_manager.cc:4841] TS 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283): tablet d3d8fd58a5264d0b99ea9af44688fd1f (table test-table [id=55843c40a51247ef9f732d1cb591d7fc]) successfully deleted
I20260924 01:03:03.229707 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15389
I20260924 01:03:03.284730 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:03.257406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.8.254.125:36617)
0924 01:03:03.257667 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:03.257676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:03.266441 (+ 8765us) server_negotiation.cc:407] Connection header received
0924 01:03:03.267117 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:03.267122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:03.267196 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:03.267261 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:03.270323 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.271137 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.273435 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.273912 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.274004 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:03.278605 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:03.278610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:03.278613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:03.278649 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:03.281664 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.282472 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.282478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.282482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.282606 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.283041 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.283045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.283048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.283410 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:03.283641 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:03.284113 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 01:03:03.284371 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:03:03.285702 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:03.257157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.8.254.124:40533)
0924 01:03:03.258576 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:03.258585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:03.266490 (+ 7905us) server_negotiation.cc:407] Connection header received
0924 01:03:03.267084 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:03.267093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:03.267181 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:03.267259 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:03.270119 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.271269 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.273404 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:03.274034 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:03.274169 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:03.278563 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:03.278572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:03.278578 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:03.278628 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:03.282148 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.283002 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.283010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.283017 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.283167 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:03.283574 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:03.283578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:03.283583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:03.283891 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:03.284143 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:03.285241 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0924 01:03:03.285479 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213}
I20260924 01:03:04.047860 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.046880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:56677)
0924 01:03:04.047423 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.047627 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
W20260924 01:03:04.048339 16545 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:38457 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.052860 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.051983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:37173)
0924 01:03:04.052491 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.052640 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":398,"threads_started":1}
W20260924 01:03:04.053386 16686 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:38457 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.054837 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.047037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:60725)
0924 01:03:04.054582 (+ 7545us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.054694 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7391,"thread_start_us":7296,"threads_started":1}
W20260924 01:03:04.055240 16397 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:38457 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.560415 16760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1bd51e6cfd7d4588a8ebdfe215842ac7)
I20260924 01:03:04.560777 16760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:04.560902 16761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1bd51e6cfd7d4588a8ebdfe215842ac7)
I20260924 01:03:04.561198 16761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:04.562251 16760 leader_election.cc:288] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457), 2130f3384151439b885c797d8eb9cc7f (127.8.254.124:40533)
I20260924 01:03:04.562685 16761 leader_election.cc:288] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457), 00805083ae6244a0876a5fda23e25e1d (127.8.254.125:36617)
I20260924 01:03:04.563851 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.562503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49536)
0924 01:03:04.563374 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.563566 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":91,"thread_start_us":559,"threads_started":1}
W20260924 01:03:04.564132 16013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.565308 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.563028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49538)
0924 01:03:04.564800 (+ 1772us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.564978 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":205,"thread_start_us":1523,"threads_started":1}
I20260924 01:03:04.567104 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.566556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49560)
0924 01:03:04.566762 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.566884 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 01:03:04.567368 16013 leader_election.cc:334] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
W20260924 01:03:04.567965 16202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.570477 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.569926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49568)
0924 01:03:04.570129 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.570256 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 01:03:04.570825 16202 leader_election.cc:334] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.573792 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.564860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.8.254.125:36617)
0924 01:03:04.566681 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:04.566689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:04.566712 (+ 23us) server_negotiation.cc:407] Connection header received
0924 01:03:04.566775 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:04.566779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:04.566860 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:04.567746 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:04.567955 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.569292 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.571650 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.572585 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.572768 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:04.572875 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:04.573213 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 01:03:04.573393 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1064,"threads_started":1}
I20260924 01:03:04.573815 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.563028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:57708)
0924 01:03:04.563729 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.563750 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:04.563886 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:04.567056 (+ 3170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:04.567063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:04.567102 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:04.567546 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:04.567563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.569464 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.569471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:04.571498 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:04.571511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.572804 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.572810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:04.573033 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:04.573061 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 01:03:04.573188 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 01:03:04.573350 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":24,"thread_start_us":403,"threads_started":1}
I20260924 01:03:04.574175 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.562499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:40533 (local address 127.0.0.1:43030)
0924 01:03:04.563037 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.563070 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:04.563229 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:04.568404 (+ 5175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:04.568412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:04.568452 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:04.568866 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:04.568880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.570588 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.570594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:04.572145 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:04.572158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.573374 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.573380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:04.573549 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:04.573575 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 01:03:04.573683 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 01:03:04.573820 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260924 01:03:04.574246 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.564987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.8.254.124:40533)
0924 01:03:04.568058 (+ 3071us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:04.568067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:04.568086 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:03:04.568147 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:04.568155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:04.568234 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:04.568346 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:04.569120 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.570381 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.572306 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:04.573166 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:04.573332 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:04.573436 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:04.573735 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 01:03:04.573865 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":2915,"thread_start_us":2781,"threads_started":1}
I20260924 01:03:04.574810 16037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2130f3384151439b885c797d8eb9cc7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00805083ae6244a0876a5fda23e25e1d" is_pre_election: true
I20260924 01:03:04.574926 16227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00805083ae6244a0876a5fda23e25e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2130f3384151439b885c797d8eb9cc7f" is_pre_election: true
I20260924 01:03:04.575201 16037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2130f3384151439b885c797d8eb9cc7f in term 1.
I20260924 01:03:04.575230 16227 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00805083ae6244a0876a5fda23e25e1d in term 1.
I20260924 01:03:04.575780 16011 leader_election.cc:302] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00805083ae6244a0876a5fda23e25e1d, 2130f3384151439b885c797d8eb9cc7f; no voters: 1bd51e6cfd7d4588a8ebdfe215842ac7
I20260924 01:03:04.575769 16201 leader_election.cc:302] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00805083ae6244a0876a5fda23e25e1d, 2130f3384151439b885c797d8eb9cc7f; no voters: 1bd51e6cfd7d4588a8ebdfe215842ac7
I20260924 01:03:04.576161 16760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 01:03:04.576184 16761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 01:03:04.576292 16760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1bd51e6cfd7d4588a8ebdfe215842ac7)
I20260924 01:03:04.576304 16761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1bd51e6cfd7d4588a8ebdfe215842ac7)
I20260924 01:03:04.576399 16761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 1 FOLLOWER]: Advancing to term 2
I20260924 01:03:04.576404 16760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 1 FOLLOWER]: Advancing to term 2
I20260924 01:03:04.578594 16761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:04.578811 16760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:04.579348 16761 leader_election.cc:288] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 2 election: Requested vote from peers 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457), 00805083ae6244a0876a5fda23e25e1d (127.8.254.125:36617)
I20260924 01:03:04.579607 16760 leader_election.cc:288] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [CANDIDATE]: Term 2 election: Requested vote from peers 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457), 2130f3384151439b885c797d8eb9cc7f (127.8.254.124:40533)
I20260924 01:03:04.579877 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.579460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49578)
0924 01:03:04.579635 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.579729 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 01:03:04.580093 16227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00805083ae6244a0876a5fda23e25e1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2130f3384151439b885c797d8eb9cc7f"
I20260924 01:03:04.580157 16037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2130f3384151439b885c797d8eb9cc7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00805083ae6244a0876a5fda23e25e1d"
I20260924 01:03:04.580392 16227 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 00805083ae6244a0876a5fda23e25e1d in current term 2: Already voted for candidate 2130f3384151439b885c797d8eb9cc7f in this term.
I20260924 01:03:04.580408 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.579776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49588)
0924 01:03:04.580084 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.580196 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260924 01:03:04.580479 16037 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2130f3384151439b885c797d8eb9cc7f in current term 2: Already voted for candidate 00805083ae6244a0876a5fda23e25e1d in this term.
I20260924 01:03:04.581538 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.581166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49596)
0924 01:03:04.581322 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.581419 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 01:03:04.581585 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:04.581241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49612)
0924 01:03:04.581395 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:04.581469 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 01:03:04.581795 16013 leader_election.cc:334] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.581921 16013 leader_election.cc:302] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00805083ae6244a0876a5fda23e25e1d; no voters: 1bd51e6cfd7d4588a8ebdfe215842ac7, 2130f3384151439b885c797d8eb9cc7f
I20260924 01:03:04.582212 16760 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 01:03:04.582334 16202 leader_election.cc:334] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:04.582507 16202 leader_election.cc:302] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2130f3384151439b885c797d8eb9cc7f; no voters: 00805083ae6244a0876a5fda23e25e1d, 1bd51e6cfd7d4588a8ebdfe215842ac7
I20260924 01:03:04.582801 16761 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 01:03:05.049710 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:05.049032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:53719)
0924 01:03:05.049457 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:05.049547 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":228,"threads_started":1}
I20260924 01:03:05.055243 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:05.054239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:36341)
0924 01:03:05.054799 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:05.054902 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":301,"threads_started":1}
I20260924 01:03:05.056417 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:05.055798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:49357)
0924 01:03:05.056188 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:05.056273 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":194,"threads_started":1}
I20260924 01:03:06.051309 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.050545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:43023)
0924 01:03:06.051003 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.051089 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":242,"threads_started":1}
I20260924 01:03:06.057093 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.056067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:45355)
0924 01:03:06.056540 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.056847 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260924 01:03:06.057976 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.057274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:35759)
0924 01:03:06.057729 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.057814 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":252,"threads_started":1}
I20260924 01:03:06.252390 16776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:06.252672 16776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:06.253407 16776 leader_election.cc:288] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457), 00805083ae6244a0876a5fda23e25e1d (127.8.254.125:36617)
I20260924 01:03:06.253995 16037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2130f3384151439b885c797d8eb9cc7f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00805083ae6244a0876a5fda23e25e1d" is_pre_election: true
I20260924 01:03:06.254315 16037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2130f3384151439b885c797d8eb9cc7f in term 2.
I20260924 01:03:06.254588 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.253661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49872)
0924 01:03:06.254226 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.254354 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
I20260924 01:03:06.254798 16201 leader_election.cc:302] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00805083ae6244a0876a5fda23e25e1d, 2130f3384151439b885c797d8eb9cc7f; no voters:
I20260924 01:03:06.255100 16776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 01:03:06.255314 16776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:06.255435 16776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 2 FOLLOWER]: Advancing to term 3
I20260924 01:03:06.256140 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.255706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49876)
0924 01:03:06.255890 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.255976 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:03:06.256340 16202 leader_election.cc:334] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:06.257797 16776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:06.258472 16776 leader_election.cc:288] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 3 election: Requested vote from peers 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457), 00805083ae6244a0876a5fda23e25e1d (127.8.254.125:36617)
I20260924 01:03:06.258880 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.258562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49884)
0924 01:03:06.258702 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.258765 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 01:03:06.259016 16037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2130f3384151439b885c797d8eb9cc7f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00805083ae6244a0876a5fda23e25e1d"
I20260924 01:03:06.259243 16037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 2 FOLLOWER]: Advancing to term 3
I20260924 01:03:06.261268 16037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2130f3384151439b885c797d8eb9cc7f in term 3.
I20260924 01:03:06.261754 16201 leader_election.cc:302] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00805083ae6244a0876a5fda23e25e1d, 2130f3384151439b885c797d8eb9cc7f; no voters:
I20260924 01:03:06.262036 16776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 01:03:06.262567 16776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [term 3 LEADER]: Becoming Leader. State: Replica: 2130f3384151439b885c797d8eb9cc7f, State: Running, Role: LEADER
I20260924 01:03:06.262678 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.262173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49888)
0924 01:03:06.262341 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.262505 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 01:03:06.262876 16202 leader_election.cc:334] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
I20260924 01:03:06.263017 16776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } }
I20260924 01:03:06.264945 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2130f3384151439b885c797d8eb9cc7f. Latest consensus state: current_term: 3 leader_uuid: "2130f3384151439b885c797d8eb9cc7f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:06.265297 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:06.265972 16781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:06.269495 16781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:06.270370 16781 catalog_manager.cc:1259] Loaded cluster ID: aafcc22a693344fc831348d8a2da7bdb
I20260924 01:03:06.270437 16781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:06.271761 16781 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:06.272637 16781 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Loaded TSK: 0
I20260924 01:03:06.273578 16781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:03:06.326741 16216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:06.334249 16216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:06.339080 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.338575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49906)
0924 01:03:06.338758 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.338893 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 01:03:06.339099 16037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [term 3 FOLLOWER]: Refusing update from remote peer 2130f3384151439b885c797d8eb9cc7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 01:03:06.339882 16779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 01:03:06.340138 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.339772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49916)
0924 01:03:06.339945 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.340011 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 01:03:06.340404 16202 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f -> Peer 1bd51e6cfd7d4588a8ebdfe215842ac7 (127.8.254.126:38457): Couldn't send request to peer 1bd51e6cfd7d4588a8ebdfe215842ac7. Status: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 01:03:06.342566 16783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2130f3384151439b885c797d8eb9cc7f. Latest consensus state: current_term: 3 leader_uuid: "2130f3384151439b885c797d8eb9cc7f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:06.342836 16783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:03:06.351219 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2130f3384151439b885c797d8eb9cc7f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:06.351558 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:06.352531 16783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2130f3384151439b885c797d8eb9cc7f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bd51e6cfd7d4588a8ebdfe215842ac7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38457 } } peers { permanent_uuid: "00805083ae6244a0876a5fda23e25e1d" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 36617 } attrs { promote: false } } peers { permanent_uuid: "2130f3384151439b885c797d8eb9cc7f" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 40533 } attrs { promote: false } } }
I20260924 01:03:06.352804 16783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00805083ae6244a0876a5fda23e25e1d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:03:06.367252 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.359936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40283 (local address 127.0.0.1:53568)
0924 01:03:06.360470 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.360488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:06.360605 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:06.361355 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:06.361360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:06.361387 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:06.361758 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:06.361769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.363257 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.363265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:06.364976 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:06.364988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.366071 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.366077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:06.366302 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.366334 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:03:06.366429 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 01:03:06.366560 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":534,"threads_started":1}
I20260924 01:03:06.367501 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.360403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.8.254.65:40283)
0924 01:03:06.361058 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:06.361063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:06.361077 (+ 14us) server_negotiation.cc:407] Connection header received
0924 01:03:06.361132 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:06.361135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:06.361188 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:06.361894 (+ 706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:06.362109 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.363123 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.365128 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.365910 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.366053 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.366159 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:06.366949 (+ 790us) server_negotiation.cc:299] Negotiation successful
0924 01:03:06.367100 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":425,"threads_started":1}
I20260924 01:03:06.368894 16331 tablet_service.cc:1467] Processing CreateTablet for tablet 9098ece0597f4903aba918c2b07aaef4 (DEFAULT_TABLE table=test-table [id=c1c10867d708418a909d12f466720bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:06.369405 16331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9098ece0597f4903aba918c2b07aaef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:06.373221 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.359601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41739 (local address 127.0.0.1:53102)
0924 01:03:06.359777 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.359797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:06.359911 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:06.366800 (+ 6889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:06.366816 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:06.366841 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:06.367448 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:06.367466 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.368722 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.368727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:06.370696 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:06.370712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.372422 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.372429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:06.372608 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.372644 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 01:03:06.372715 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 01:03:06.372871 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":7}
I20260924 01:03:06.373823 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.362370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:32783 (local address 127.0.0.1:39010)
0924 01:03:06.363047 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.363070 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:06.363210 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:06.367056 (+ 3846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:06.367061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:06.367088 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:06.368581 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:06.368596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.369795 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.369803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:06.371446 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:06.371458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.373047 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.373052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:06.373177 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.373200 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 01:03:06.373269 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 01:03:06.373560 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":990,"thread_start_us":317,"threads_started":1}
I20260924 01:03:06.374135 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.359725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.8.254.66:41739)
0924 01:03:06.366490 (+ 6765us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:06.366497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:06.366514 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:03:06.366578 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:06.366583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:06.366655 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:06.366760 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:06.367645 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.368523 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.370922 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.372171 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.373486 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.373590 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:06.373644 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 01:03:06.373787 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6606,"thread_start_us":2499,"threads_started":1}
I20260924 01:03:06.374778 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.362437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.8.254.67:32783)
0924 01:03:06.366672 (+ 4235us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:06.366676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:06.366691 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:06.366780 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:06.366786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:06.366859 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:06.367018 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:06.368803 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.369666 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.371600 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.372808 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.374161 (+ 1353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.374271 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:06.374330 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 01:03:06.374461 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4126,"thread_start_us":3999,"threads_started":1}
I20260924 01:03:06.374908 16479 tablet_service.cc:1467] Processing CreateTablet for tablet 9098ece0597f4903aba918c2b07aaef4 (DEFAULT_TABLE table=test-table [id=c1c10867d708418a909d12f466720bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:06.375370 16479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9098ece0597f4903aba918c2b07aaef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:06.375499 16620 tablet_service.cc:1467] Processing CreateTablet for tablet 9098ece0597f4903aba918c2b07aaef4 (DEFAULT_TABLE table=test-table [id=c1c10867d708418a909d12f466720bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:06.375913 16620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9098ece0597f4903aba918c2b07aaef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:06.376456 16794 tablet_bootstrap.cc:492] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Bootstrap starting.
I20260924 01:03:06.378187 16794 tablet_bootstrap.cc:654] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:06.380520 16794 tablet_bootstrap.cc:492] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: No bootstrap required, opened a new log
I20260924 01:03:06.380645 16794 ts_tablet_manager.cc:1397] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 01:03:06.381407 16794 raft_consensus.cc:348] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.381697 16794 raft_consensus.cc:374] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:06.381793 16794 raft_consensus.cc:729] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376712ccc3674ae3be2916b277ef8a19, State: Initialized, Role: FOLLOWER
I20260924 01:03:06.382074 16797 tablet_bootstrap.cc:492] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Bootstrap starting.
I20260924 01:03:06.382092 16794 consensus_queue.cc:260] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.382999 16794 ts_tablet_manager.cc:1428] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 01:03:06.384161 16797 tablet_bootstrap.cc:654] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:06.384286 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.383601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:58757)
0924 01:03:06.384012 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.384116 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290}
I20260924 01:03:06.385344 16796 tablet_bootstrap.cc:492] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Bootstrap starting.
I20260924 01:03:06.385538 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.385082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:56067)
0924 01:03:06.385297 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.385390 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 01:03:06.386384 16396 heartbeater.cc:503] Master 127.8.254.124:40533 was elected leader, sending a full tablet report...
I20260924 01:03:06.387135 16796 tablet_bootstrap.cc:654] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:06.389061 16797 tablet_bootstrap.cc:492] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: No bootstrap required, opened a new log
I20260924 01:03:06.389154 16796 tablet_bootstrap.cc:492] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: No bootstrap required, opened a new log
I20260924 01:03:06.389212 16797 ts_tablet_manager.cc:1397] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260924 01:03:06.389289 16796 ts_tablet_manager.cc:1397] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 01:03:06.389905 16797 raft_consensus.cc:348] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.389909 16796 raft_consensus.cc:348] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.390076 16797 raft_consensus.cc:374] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:06.390115 16796 raft_consensus.cc:374] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:06.390211 16797 raft_consensus.cc:729] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fef41032fb241dd8d8c1bcdf5d09396, State: Initialized, Role: FOLLOWER
I20260924 01:03:06.390234 16796 raft_consensus.cc:729] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b7ef6ee4f341db9e79fb6952e03a83, State: Initialized, Role: FOLLOWER
I20260924 01:03:06.390477 16797 consensus_queue.cc:260] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.390496 16796 consensus_queue.cc:260] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.391278 16797 ts_tablet_manager.cc:1428] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 01:03:06.391381 16796 ts_tablet_manager.cc:1428] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 01:03:06.392343 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.391814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:46665)
0924 01:03:06.392105 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.392204 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260924 01:03:06.392428 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.392038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:48379)
0924 01:03:06.392198 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.392284 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 01:03:06.395022 16544 heartbeater.cc:503] Master 127.8.254.124:40533 was elected leader, sending a full tablet report...
I20260924 01:03:06.395218 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.394718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:41993)
0924 01:03:06.394940 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.395037 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 01:03:06.396201 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.395722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:56345)
0924 01:03:06.395928 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.396016 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 01:03:06.396726 16685 heartbeater.cc:503] Master 127.8.254.124:40533 was elected leader, sending a full tablet report...
I20260924 01:03:06.799176 16802 raft_consensus.cc:482] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:06.799459 16802 raft_consensus.cc:504] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.800180 16802 leader_election.cc:288] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283), 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:06.800804 16640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098ece0597f4903aba918c2b07aaef4" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" is_pre_election: true
I20260924 01:03:06.800800 16350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098ece0597f4903aba918c2b07aaef4" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376712ccc3674ae3be2916b277ef8a19" is_pre_election: true
I20260924 01:03:06.801060 16640 raft_consensus.cc:2454] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 0.
I20260924 01:03:06.801103 16350 raft_consensus.cc:2454] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 0.
I20260924 01:03:06.801573 16435 leader_election.cc:302] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fef41032fb241dd8d8c1bcdf5d09396, a2b7ef6ee4f341db9e79fb6952e03a83; no voters:
I20260924 01:03:06.801925 16802 raft_consensus.cc:2781] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:06.802057 16802 raft_consensus.cc:482] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:06.802157 16802 raft_consensus.cc:3037] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:06.803043 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.802631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49920)
0924 01:03:06.802805 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.802910 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 01:03:06.804061 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.803689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49930)
0924 01:03:06.803848 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.803930 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 01:03:06.804603 16802 raft_consensus.cc:504] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.805137 16802 leader_election.cc:288] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 election: Requested vote from peers 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283), 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783)
I20260924 01:03:06.805673 16640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098ece0597f4903aba918c2b07aaef4" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fef41032fb241dd8d8c1bcdf5d09396"
I20260924 01:03:06.805893 16640 raft_consensus.cc:3037] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:06.806073 16350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9098ece0597f4903aba918c2b07aaef4" candidate_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376712ccc3674ae3be2916b277ef8a19"
I20260924 01:03:06.806325 16350 raft_consensus.cc:3037] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:06.809005 16640 raft_consensus.cc:2454] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 1.
I20260924 01:03:06.809293 16350 raft_consensus.cc:2454] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b7ef6ee4f341db9e79fb6952e03a83 in term 1.
I20260924 01:03:06.809511 16435 leader_election.cc:302] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fef41032fb241dd8d8c1bcdf5d09396, a2b7ef6ee4f341db9e79fb6952e03a83; no voters:
I20260924 01:03:06.809816 16802 raft_consensus.cc:2781] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:06.810312 16802 raft_consensus.cc:686] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 LEADER]: Becoming Leader. State: Replica: a2b7ef6ee4f341db9e79fb6952e03a83, State: Running, Role: LEADER
I20260924 01:03:06.810621 16802 consensus_queue.cc:237] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } }
I20260924 01:03:06.812945 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.812491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:56297)
0924 01:03:06.812691 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.812763 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 01:03:06.813942 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.813557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:49347)
0924 01:03:06.813751 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.813812 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 01:03:06.813889 16216 catalog_manager.cc:5495] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b7ef6ee4f341db9e79fb6952e03a83" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 } health_report { overall_health: UNKNOWN } } }
I20260924 01:03:06.939109 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.933846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.8.254.66:41739)
0924 01:03:06.934431 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:06.934435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:06.934453 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:03:06.934577 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:06.934581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:06.934646 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:06.934730 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:06.935135 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.935815 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.937271 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.937867 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.937983 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:06.938365 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:06.938512 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:06.938613 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:06.938712 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 01:03:06.938794 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":361,"threads_started":1}
I20260924 01:03:06.942338 16350 raft_consensus.cc:1261] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Refusing update from remote peer a2b7ef6ee4f341db9e79fb6952e03a83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:06.942328 16640 raft_consensus.cc:1261] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Refusing update from remote peer a2b7ef6ee4f341db9e79fb6952e03a83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:06.942958 16803 consensus_queue.cc:1035] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fef41032fb241dd8d8c1bcdf5d09396" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:06.943270 16802 consensus_queue.cc:1035] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "376712ccc3674ae3be2916b277ef8a19" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:06.947113 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.945505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:46797)
0924 01:03:06.946083 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.946231 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"threads_started":1}
I20260924 01:03:06.955229 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.946088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:42191)
0924 01:03:06.954839 (+ 8751us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.954981 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8554,"thread_start_us":8446,"threads_started":1}
I20260924 01:03:06.959893 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.959232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.65:37299)
0924 01:03:06.959578 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.959688 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260924 01:03:06.961112 16814 mvcc.cc:205] Tried to move back new op lower bound from 7332707479309705216 to 7332707478778531840. Current Snapshot: MvccSnapshot[applied={T|T < 7332707479309705216}]
I20260924 01:03:06.963110 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.962495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.67:60235)
0924 01:03:06.962728 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.962849 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 01:03:06.974001 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.973441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:45069)
0924 01:03:06.973724 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.973820 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 01:03:06.975204 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.974751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.8.254.66:34579)
0924 01:03:06.974965 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:06.975044 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 01:03:06.977696 16216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:06.977952 16216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:06.983055 16216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2130f3384151439b885c797d8eb9cc7f: Sending DeleteTablet for 3 replicas of tablet 9098ece0597f4903aba918c2b07aaef4
I20260924 01:03:06.984280 16479 tablet_service.cc:1514] Processing DeleteTablet for tablet 9098ece0597f4903aba918c2b07aaef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:06 UTC) from {username='oryx', principal='oryx/127.8.254.124@KRBTEST.COM'} at 127.0.0.1:53102
I20260924 01:03:06.984393 16331 tablet_service.cc:1514] Processing DeleteTablet for tablet 9098ece0597f4903aba918c2b07aaef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:06 UTC) from {username='oryx', principal='oryx/127.8.254.124@KRBTEST.COM'} at 127.0.0.1:53568
I20260924 01:03:06.984464 16620 tablet_service.cc:1514] Processing DeleteTablet for tablet 9098ece0597f4903aba918c2b07aaef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:06 UTC) from {username='oryx', principal='oryx/127.8.254.124@KRBTEST.COM'} at 127.0.0.1:39010
I20260924 01:03:06.985219 16822 tablet_replica.cc:330] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: stopping tablet replica
I20260924 01:03:06.985530 16822 raft_consensus.cc:2229] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:06.985797 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 16204
I20260924 01:03:06.986032 16822 raft_consensus.cc:2258] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:06.991042 16822 ts_tablet_manager.cc:1905] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:06.991783 16823 tablet_replica.cc:330] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: stopping tablet replica
I20260924 01:03:06.992125 16823 raft_consensus.cc:2229] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:06.992532 16823 raft_consensus.cc:2258] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:06.993835 16823 ts_tablet_manager.cc:1905] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:06.994397 16821 tablet_replica.cc:330] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: stopping tablet replica
I20260924 01:03:06.994599 16821 raft_consensus.cc:2229] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:06.994994 16821 raft_consensus.cc:2258] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:06.996079 16821 ts_tablet_manager.cc:1905] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:07.000461 16822 ts_tablet_manager.cc:1918] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:07.000628 16822 log.cc:1196] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/9098ece0597f4903aba918c2b07aaef4
I20260924 01:03:07.000692 16823 ts_tablet_manager.cc:1918] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:07.000831 16823 log.cc:1196] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/9098ece0597f4903aba918c2b07aaef4
I20260924 01:03:07.001143 16822 ts_tablet_manager.cc:1939] T 9098ece0597f4903aba918c2b07aaef4 P 376712ccc3674ae3be2916b277ef8a19: Deleting consensus metadata
I20260924 01:03:07.001251 16823 ts_tablet_manager.cc:1939] T 9098ece0597f4903aba918c2b07aaef4 P 3fef41032fb241dd8d8c1bcdf5d09396: Deleting consensus metadata
I20260924 01:03:07.002485 16202 catalog_manager.cc:4841] TS 376712ccc3674ae3be2916b277ef8a19 (127.8.254.65:40283): tablet 9098ece0597f4903aba918c2b07aaef4 (table test-table [id=c1c10867d708418a909d12f466720bcc]) successfully deleted
I20260924 01:03:07.002483 16203 catalog_manager.cc:4841] TS 3fef41032fb241dd8d8c1bcdf5d09396 (127.8.254.67:32783): tablet 9098ece0597f4903aba918c2b07aaef4 (table test-table [id=c1c10867d708418a909d12f466720bcc]) successfully deleted
I20260924 01:03:07.003722 16821 ts_tablet_manager.cc:1918] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:07.003890 16821 log.cc:1196] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/9098ece0597f4903aba918c2b07aaef4
I20260924 01:03:07.004340 16821 ts_tablet_manager.cc:1939] T 9098ece0597f4903aba918c2b07aaef4 P a2b7ef6ee4f341db9e79fb6952e03a83: Deleting consensus metadata
I20260924 01:03:07.005573 16200 catalog_manager.cc:4841] TS a2b7ef6ee4f341db9e79fb6952e03a83 (127.8.254.66:41739): tablet 9098ece0597f4903aba918c2b07aaef4 (table test-table [id=c1c10867d708418a909d12f466720bcc]) successfully deleted
I20260924 01:03:07.009672 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:06.986633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.8.254.65:40283)
0924 01:03:06.993363 (+ 6730us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:06.993369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:06.993397 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:03:06.993465 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:06.993470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:06.993563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:06.993751 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:06.995632 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.996695 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.998673 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:06.999410 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:06.999550 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:07.003527 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:07.003543 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:07.003560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:07.003595 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:07.007120 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:07.007806 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:07.007814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:07.007817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:07.007948 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:07.008294 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:07.008297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:07.008299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:07.008648 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:07.008876 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:07.009127 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 01:03:07.009291 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6543,"thread_start_us":6420,"threads_started":1}
I20260924 01:03:07.010462 16391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:07.104107 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 16204
I20260924 01:03:07.121805 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 16408
I20260924 01:03:07.135171 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:07.122644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.8.254.66:41739)
0924 01:03:07.122826 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:07.122831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:07.122852 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:03:07.122995 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:07.122999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:07.123069 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:07.123188 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:07.124553 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:07.125262 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:07.126600 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:07.126993 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:07.127107 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:07.129670 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:07.129687 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:07.129704 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:07.129737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:07.132432 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:07.133064 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:07.133071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:07.133073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:07.133300 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:07.133656 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:07.133660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:07.133662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:07.134002 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:07.134467 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:07.134598 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 01:03:07.134816 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:03:07.135845 16539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:07.257843 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 16408
I20260924 01:03:07.279817 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 16557
I20260924 01:03:07.294240 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:07.280677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.8.254.67:32783)
0924 01:03:07.281268 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:07.281273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:07.281288 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:07.281342 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:07.281345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:07.281408 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:07.281529 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:07.283066 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:07.283808 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:07.285030 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:07.285446 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:07.285582 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:07.288245 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:07.288262 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:07.288279 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:07.288314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:07.291296 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:07.292003 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:07.292011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:07.292013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:07.292163 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:07.292590 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:07.292595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:07.292597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:07.292949 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:07.293298 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:07.293621 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 01:03:07.293851 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
I20260924 01:03:07.294972 16680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:07.389954 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 16557
I20260924 01:03:07.408448 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15883
I20260924 01:03:07.409711 16057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:07.416992 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:07.416303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49940)
0924 01:03:07.416729 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:07.416809 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260924 01:03:07.418426 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:07.418017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38457 (local address 127.0.0.1:49954)
0924 01:03:07.418157 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:07.418278 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 01:03:07.532195 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15883
I20260924 01:03:07.553937 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 15982
I20260924 01:03:07.555030 16247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:07.680521 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:07.679963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:36617 (local address 127.0.0.1:57722)
0924 01:03:07.680148 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:07.680270 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:36617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 01:03:07.680522 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 15982
2026-09-24T01:03:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12113 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:03:07 dist-test-slave-5c7l krb5kdc[16836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:03:07 dist-test-slave-5c7l krb5kdc[16836](info): set up 2 sockets
Sep 24 01:03:07 dist-test-slave-5c7l krb5kdc[16836](info): commencing operation
krb5kdc: starting...
W20260924 01:03:08.804188 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:03:08 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab.
Entry for principal oryx/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab.
I20260924 01:03:09.026422 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:43107
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab
--principal=oryx/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:43107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:09.353439 16849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:09.353749 16849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:09.353912 16849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:09.361292 16849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:09.361413 16849 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 01:03:09.361496 16849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:09.361596 16849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:09.361687 16849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:09.361802 16849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:09.371974 16849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:43107
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.126.keytab
--principal=oryx/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:09.373369 16849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:09.374810 16849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:09.386741 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:09.386775 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:09.386798 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:09.387547 16849 server_base.cc:1034] running on GCE node
I20260924 01:03:09.388412 16849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:09.389825 16849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:09.391077 16849 hybrid_clock.cc:648] HybridClock initialized: now 1790211789391024 us; error 42 us; skew 500 ppm
I20260924 01:03:09.391448 16849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:09 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211789, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:09.394548 16849 init.cc:377] Logged in from keytab as oryx/127.8.254.126@KRBTEST.COM (short username oryx)
I20260924 01:03:09.395973 16849 webserver.cc:460] Webserver started at http://127.8.254.126:35591/ using document root <none> and password file <none>
I20260924 01:03:09.396677 16849 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:09.396811 16849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:09.397151 16849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:09.400156 16849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "1929b03b7a4744bc9c74f392644e5797"
format_stamp: "Formatted at 2026-09-24 01:03:09 on dist-test-slave-5c7l"
server_key: "fd215a1f49f7b4c5a178245ac6a727e6"
server_key_iv: "b612834ff51d51db5bd6914393d4dc18"
server_key_version: "encryptionkey@0"
I20260924 01:03:09.400985 16849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "1929b03b7a4744bc9c74f392644e5797"
format_stamp: "Formatted at 2026-09-24 01:03:09 on dist-test-slave-5c7l"
server_key: "fd215a1f49f7b4c5a178245ac6a727e6"
server_key_iv: "b612834ff51d51db5bd6914393d4dc18"
server_key_version: "encryptionkey@0"
I20260924 01:03:09.406147 16849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:03:09.409597 16864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:09.410956 16849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:09.411142 16849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "1929b03b7a4744bc9c74f392644e5797"
format_stamp: "Formatted at 2026-09-24 01:03:09 on dist-test-slave-5c7l"
server_key: "fd215a1f49f7b4c5a178245ac6a727e6"
server_key_iv: "b612834ff51d51db5bd6914393d4dc18"
server_key_version: "encryptionkey@0"
I20260924 01:03:09.411369 16849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:09.428524 16849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:09.432775 16849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:09.433084 16849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:09.455489 16849 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:43107
I20260924 01:03:09.455507 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:43107 every 8 connection(s)
I20260924 01:03:09.456897 16849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:03:09.457573 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 16849
I20260924 01:03:09.457861 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:03:09.458702 9209 external_mini_cluster.cc:1428] Setting key d70b703563dd9eef8b520e70ec8d0dcc
I20260924 01:03:09.461560 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 01:03:09 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:09.474503 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Bootstrap starting.
I20260924 01:03:09.478770 16926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:09.479962 16926 log.cc:824] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:09.482944 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:09.460755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.8.254.126:43107)
0924 01:03:09.461588 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:09.461606 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:09.461660 (+ 54us) server_negotiation.cc:407] Connection header received
0924 01:03:09.463962 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:09.463989 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:09.464496 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:09.464962 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:09.466260 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:09.468323 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:09.469590 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:09.470634 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:09.470851 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:09.474745 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:09.474756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:09.474775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:09.474833 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:09.478831 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:09.479507 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:09.479515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:09.479525 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:09.479738 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:09.480127 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:09.480133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:09.480136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:09.480854 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:09.481169 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:09.481530 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 01:03:09.481900 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":343,"threads_started":1}
I20260924 01:03:09.483166 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: No bootstrap required, opened a new log
I20260924 01:03:09.494467 16926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1929b03b7a4744bc9c74f392644e5797" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43107 } }
I20260924 01:03:09.494982 16926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:09.495126 16926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1929b03b7a4744bc9c74f392644e5797, State: Initialized, Role: FOLLOWER
I20260924 01:03:09.495833 16926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1929b03b7a4744bc9c74f392644e5797" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43107 } }
I20260924 01:03:09.496126 16926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:03:09.496269 16926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:03:09.496479 16926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:09.499627 16926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1929b03b7a4744bc9c74f392644e5797" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43107 } }
I20260924 01:03:09.500248 16926 leader_election.cc:302] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1929b03b7a4744bc9c74f392644e5797; no voters:
I20260924 01:03:09.501032 16926 leader_election.cc:288] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:03:09.501263 16931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:09.502115 16931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [term 1 LEADER]: Becoming Leader. State: Replica: 1929b03b7a4744bc9c74f392644e5797, State: Running, Role: LEADER
I20260924 01:03:09.502995 16931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1929b03b7a4744bc9c74f392644e5797" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43107 } }
I20260924 01:03:09.503513 16926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:09.506330 16933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1929b03b7a4744bc9c74f392644e5797. Latest consensus state: current_term: 1 leader_uuid: "1929b03b7a4744bc9c74f392644e5797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1929b03b7a4744bc9c74f392644e5797" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43107 } } }
I20260924 01:03:09.506399 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1929b03b7a4744bc9c74f392644e5797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1929b03b7a4744bc9c74f392644e5797" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43107 } } }
I20260924 01:03:09.506699 16933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:09.506783 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:09.508555 16939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:09.513423 16939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:09.524391 16939 catalog_manager.cc:1347] Generated new cluster ID: 69badb8064484d6d87ee9b35028accd5
I20260924 01:03:09.524562 16939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:09.553117 16939 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:03:09.555135 16939 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:09.561882 16939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Generated new TSK 0
I20260924 01:03:09.562803 16939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab.
Entry for principal oryx/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab.
I20260924 01:03:09.625699 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:43107
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:09.959021 16952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:09.959308 16952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:09.959404 16952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:09.966550 16952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:09.966660 16952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:09.966743 16952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:03:09.966823 16952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:09.976986 16952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:09.978245 16952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:09.979526 16952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:09.992601 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:09.992642 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:09.992686 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:09.993752 16952 server_base.cc:1034] running on GCE node
I20260924 01:03:09.994444 16952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:09.995420 16952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:09.996675 16952 hybrid_clock.cc:648] HybridClock initialized: now 1790211789996582 us; error 61 us; skew 500 ppm
I20260924 01:03:09.997020 16952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:09 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211789, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:10.000564 16952 init.cc:377] Logged in from keytab as oryx/127.8.254.65@KRBTEST.COM (short username oryx)
I20260924 01:03:10.002077 16952 webserver.cc:460] Webserver started at http://127.8.254.65:36747/ using document root <none> and password file <none>
I20260924 01:03:10.002884 16952 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:10.002978 16952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:10.003257 16952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:10.006724 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "3cc0179666914f528cb2911b53d87be1"
format_stamp: "Formatted at 2026-09-24 01:03:10 on dist-test-slave-5c7l"
server_key: "b93c04f942f14424a9c760370cc21e2c"
server_key_iv: "0019efda9c0c122a11b96f6c24e9facc"
server_key_version: "encryptionkey@0"
I20260924 01:03:10.007535 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "3cc0179666914f528cb2911b53d87be1"
format_stamp: "Formatted at 2026-09-24 01:03:10 on dist-test-slave-5c7l"
server_key: "b93c04f942f14424a9c760370cc21e2c"
server_key_iv: "0019efda9c0c122a11b96f6c24e9facc"
server_key_version: "encryptionkey@0"
I20260924 01:03:10.013044 16952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:10.017045 16967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:10.018563 16952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:10.018750 16952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "3cc0179666914f528cb2911b53d87be1"
format_stamp: "Formatted at 2026-09-24 01:03:10 on dist-test-slave-5c7l"
server_key: "b93c04f942f14424a9c760370cc21e2c"
server_key_iv: "0019efda9c0c122a11b96f6c24e9facc"
server_key_version: "encryptionkey@0"
I20260924 01:03:10.018983 16952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:10.035643 16952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:10.039916 16952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:10.040225 16952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:10.043090 16952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:10.043191 16952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:10.043464 16952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:10.043568 16952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:03:10 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211789, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:10.068041 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.043656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:51325 (local address 127.8.254.126:43107)
0924 01:03:10.044793 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:10.044801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:10.045355 (+ 554us) server_negotiation.cc:407] Connection header received
0924 01:03:10.047435 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:10.047440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:10.047532 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:10.047661 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:10.049845 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.050838 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.054314 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.054935 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.055085 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.059963 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:10.059970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:10.059976 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:10.060019 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:10.064089 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.065433 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.065442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.065448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.065617 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.066179 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.066185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.066189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.066521 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:10.066682 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:10.067418 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 01:03:10.067644 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":899,"threads_started":1}
I20260924 01:03:10.069123 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.044332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.8.254.65:51325)
0924 01:03:10.045192 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:10.045234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:10.047150 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:10.047937 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:10.047951 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:10.048808 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:10.049660 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.049680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.051030 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.051038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.054134 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.054149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.055180 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.055189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.055428 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.056132 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:10.056156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:10.059622 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:10.064317 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.064336 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.064362 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.065214 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.065770 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.065777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.065780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.066039 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.066655 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:10.066663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:10.067047 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 01:03:10.067320 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 01:03:10.067885 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":389,"threads_started":1}
I20260924 01:03:10.095928 16952 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:40723
I20260924 01:03:10.095975 17087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:40723 every 8 connection(s)
I20260924 01:03:10.097662 16952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:10.100018 17088 heartbeater.cc:347] Connected to a master server at 127.8.254.126:43107
I20260924 01:03:10.100260 17088 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:10.101019 17088 heartbeater.cc:511] Master 127.8.254.126:43107 requested a full tablet report, sending...
I20260924 01:03:10.101748 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.086725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:51261 (local address 127.8.254.126:43107)
0924 01:03:10.086915 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:10.086940 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:10.087539 (+ 599us) server_negotiation.cc:407] Connection header received
0924 01:03:10.087740 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:10.087749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:10.087870 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:10.087989 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:10.089072 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.090364 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.092216 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.093003 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.093141 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.095219 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:10.095226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:10.095230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:10.095267 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:10.098804 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.099652 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.099660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.099665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.099839 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.100196 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.100202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.100205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.100543 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:10.100708 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:10.101155 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 01:03:10.101376 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:03:10.101847 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.086753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.8.254.65:51261)
0924 01:03:10.087369 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:10.087391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:10.087571 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:10.088057 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:10.088063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:10.088603 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:10.088901 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.088911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.090526 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.090534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.092051 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.092064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.093260 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.093267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.093392 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.094071 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:10.094103 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:10.095011 (+ 908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:10.098954 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.098959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.098964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.099527 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.099936 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.099941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.099945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.100083 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.100665 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:10.100671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:10.100890 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 01:03:10.101060 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 01:03:10.101381 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":5,"thread_start_us":324,"threads_started":1}
I20260924 01:03:10.103178 16881 ts_manager.cc:194] Registered new tserver with Master: 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723)
I20260924 01:03:10.105003 16881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:51325
I20260924 01:03:10.107755 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 16952
I20260924 01:03:10.107976 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:03:10.108589 9209 external_mini_cluster.cc:1428] Setting key 93162ed368db6e0e83ed4a1d26e83406
WARNING: no policy specified for oryx/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab.
Entry for principal oryx/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab.
I20260924 01:03:10.155515 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:43107
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:10.481714 17092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:10.481992 17092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:10.482089 17092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:10.489428 17092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:10.489543 17092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:10.489629 17092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:03:10.489709 17092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:10.499328 17092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:10.500524 17092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:10.501783 17092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:10.515269 17097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:10.515249 17098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:10.515468 17100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:10.516137 17092 server_base.cc:1034] running on GCE node
I20260924 01:03:10.516759 17092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:10.517633 17092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:10.518846 17092 hybrid_clock.cc:648] HybridClock initialized: now 1790211790518777 us; error 40 us; skew 500 ppm
I20260924 01:03:10.519147 17092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:10 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211790, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:10.522131 17092 init.cc:377] Logged in from keytab as oryx/127.8.254.66@KRBTEST.COM (short username oryx)
I20260924 01:03:10.523638 17092 webserver.cc:460] Webserver started at http://127.8.254.66:46639/ using document root <none> and password file <none>
I20260924 01:03:10.524272 17092 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:10.524356 17092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:10.524621 17092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:10.527542 17092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "ce84215dc674449786297ef47b19a409"
format_stamp: "Formatted at 2026-09-24 01:03:10 on dist-test-slave-5c7l"
server_key: "0cb20a52a2302384d3bde87d865c22d0"
server_key_iv: "cc8468a1726e205de231f79e2acbd3bf"
server_key_version: "encryptionkey@0"
I20260924 01:03:10.528394 17092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "ce84215dc674449786297ef47b19a409"
format_stamp: "Formatted at 2026-09-24 01:03:10 on dist-test-slave-5c7l"
server_key: "0cb20a52a2302384d3bde87d865c22d0"
server_key_iv: "cc8468a1726e205de231f79e2acbd3bf"
server_key_version: "encryptionkey@0"
I20260924 01:03:10.533521 17092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:10.537146 17107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:10.538576 17092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:10.538712 17092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "ce84215dc674449786297ef47b19a409"
format_stamp: "Formatted at 2026-09-24 01:03:10 on dist-test-slave-5c7l"
server_key: "0cb20a52a2302384d3bde87d865c22d0"
server_key_iv: "cc8468a1726e205de231f79e2acbd3bf"
server_key_version: "encryptionkey@0"
I20260924 01:03:10.538897 17092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:10.551203 17092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:10.554986 17092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:10.555207 17092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:10.557861 17092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:10.557978 17092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:10.558144 17092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:10.558235 17092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 01:03:10 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211790, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:10.586234 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.558987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.8.254.66:52755)
0924 01:03:10.559853 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:10.559892 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:10.561878 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:10.562777 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:10.562786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:10.563674 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:10.564648 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.564673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.566058 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.566068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.568884 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.568903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.570048 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.570056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.570346 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.571213 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:10.571245 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:10.575136 (+ 3891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:10.579774 (+ 4638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.579789 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.579811 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.580618 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.581156 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.581165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.581171 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.581541 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.582396 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:10.582411 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:10.583026 (+ 615us) client_negotiation.cc:769] Sending connection context
0924 01:03:10.583422 (+ 396us) client_negotiation.cc:240] Negotiation successful
0924 01:03:10.584152 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":443,"threads_started":1}
I20260924 01:03:10.587180 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.558341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:52755 (local address 127.8.254.126:43107)
0924 01:03:10.558560 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:10.558567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:10.560027 (+ 1460us) server_negotiation.cc:407] Connection header received
0924 01:03:10.562283 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:10.562289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:10.562396 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:10.562518 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:10.564838 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.565827 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.569081 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.569717 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.569872 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.575467 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:10.575476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:10.575483 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:10.575542 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:10.579552 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.580798 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.580807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.580811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.580992 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.581732 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.581740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.581746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.582176 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:10.582399 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:10.585831 (+ 3432us) server_negotiation.cc:299] Negotiation successful
0924 01:03:10.586147 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:03:10.598417 17092 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:43939
I20260924 01:03:10.598464 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:43939 every 8 connection(s)
I20260924 01:03:10.599888 17092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:10.602335 17227 heartbeater.cc:347] Connected to a master server at 127.8.254.126:43107
I20260924 01:03:10.602711 17227 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:10.603554 17227 heartbeater.cc:511] Master 127.8.254.126:43107 requested a full tablet report, sending...
I20260924 01:03:10.605291 16881 ts_manager.cc:194] Registered new tserver with Master: ce84215dc674449786297ef47b19a409 (127.8.254.66:43939)
I20260924 01:03:10.606519 16881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:52755
I20260924 01:03:10.607426 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 17092
I20260924 01:03:10.607630 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:03:10.608050 9209 external_mini_cluster.cc:1428] Setting key 26982078881a09aef997c257ac7608fa
I20260924 01:03:10.610719 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.595233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:44771 (local address 127.8.254.126:43107)
0924 01:03:10.595426 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:10.595432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:10.595703 (+ 271us) server_negotiation.cc:407] Connection header received
0924 01:03:10.595920 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:10.595924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:10.596010 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:10.596132 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:10.597270 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.598469 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.599852 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.600544 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.600704 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.603227 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:10.603236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:10.603243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:10.603284 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:10.607082 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.608328 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.608334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.608339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.608511 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:10.609007 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:10.609013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:10.609016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:10.609475 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:10.609637 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:10.610161 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 01:03:10.610420 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 01:03:10.610849 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:10.594675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.8.254.66:44771)
0924 01:03:10.595566 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:10.595593 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:10.595787 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:10.596173 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:10.596180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:10.596737 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:10.597112 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.597124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.598694 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.598700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.599708 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:10.599720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:10.600776 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:10.600783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:10.600982 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:10.601859 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:10.601924 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:10.603026 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:10.607653 (+ 4627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.607658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.607662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.608190 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.608670 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:10.608675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:10.608678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:10.608843 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:10.609625 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:10.609630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:10.609862 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 01:03:10.610033 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 01:03:10.610398 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":3,"thread_start_us":605,"threads_started":1}
WARNING: no policy specified for oryx/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab.
Entry for principal oryx/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab.
I20260924 01:03:10.657145 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:43107
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:10.982530 17231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:10.982858 17231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:10.983036 17231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:10.990648 17231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:10.990798 17231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:10.990939 17231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 01:03:10.991070 17231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:11.001184 17231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.254.126:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:11.002725 17231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:11.004222 17231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:11.018118 17236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:11.018147 17237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:11.018751 17239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:11.019057 17231 server_base.cc:1034] running on GCE node
I20260924 01:03:11.019655 17231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:11.020563 17231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:11.021776 17231 hybrid_clock.cc:648] HybridClock initialized: now 1790211791021689 us; error 61 us; skew 500 ppm
I20260924 01:03:11.022102 17231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:11 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211791, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:11.025396 17231 init.cc:377] Logged in from keytab as oryx/127.8.254.67@KRBTEST.COM (short username oryx)
I20260924 01:03:11.026986 17231 webserver.cc:460] Webserver started at http://127.8.254.67:41831/ using document root <none> and password file <none>
I20260924 01:03:11.027650 17231 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:11.027757 17231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:11.028039 17231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:11.031031 17231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "c77662a1a990439eacb482b49021fa43"
format_stamp: "Formatted at 2026-09-24 01:03:11 on dist-test-slave-5c7l"
server_key: "b0f88364d05542c17589c365fc8d139b"
server_key_iv: "746aedeb20555fda4c4afb2a9dfe1755"
server_key_version: "encryptionkey@0"
I20260924 01:03:11.031785 17231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "c77662a1a990439eacb482b49021fa43"
format_stamp: "Formatted at 2026-09-24 01:03:11 on dist-test-slave-5c7l"
server_key: "b0f88364d05542c17589c365fc8d139b"
server_key_iv: "746aedeb20555fda4c4afb2a9dfe1755"
server_key_version: "encryptionkey@0"
I20260924 01:03:11.037042 17231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:11.040551 17246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:11.041939 17231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:03:11.042150 17231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "c77662a1a990439eacb482b49021fa43"
format_stamp: "Formatted at 2026-09-24 01:03:11 on dist-test-slave-5c7l"
server_key: "b0f88364d05542c17589c365fc8d139b"
server_key_iv: "746aedeb20555fda4c4afb2a9dfe1755"
server_key_version: "encryptionkey@0"
I20260924 01:03:11.042451 17231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:11.082929 17231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:11.087481 17231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:11.087773 17231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:11.090663 17231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:11.090775 17231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:11.090936 17231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:11.091028 17231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:03:11 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211791, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:11.107756 17088 heartbeater.cc:503] Master 127.8.254.126:43107 was elected leader, sending a full tablet report...
I20260924 01:03:11.114635 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:11.091480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:51399 (local address 127.8.254.126:43107)
0924 01:03:11.091720 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:11.091729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:11.093479 (+ 1750us) server_negotiation.cc:407] Connection header received
0924 01:03:11.096126 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:11.096133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:11.096228 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:11.096298 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:11.098572 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.099713 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.102028 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.102881 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.103047 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:11.108238 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:11.108243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:11.108247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:11.108280 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:11.111454 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:11.112181 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:11.112186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:11.112192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:11.112314 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:11.112760 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:11.112765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:11.112768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:11.113032 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:11.113188 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:11.113969 (+ 781us) server_negotiation.cc:299] Negotiation successful
0924 01:03:11.114218 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:03:11.116402 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:11.092511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.8.254.67:51399)
0924 01:03:11.093320 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:11.093357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:11.095757 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:11.096800 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:11.096811 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:11.097521 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:11.098328 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:11.098353 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.099963 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.099968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:11.101863 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:11.101883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.103196 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.103206 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:11.103525 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:11.104404 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:11.104445 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:11.108022 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:11.111592 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:11.111601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:11.111616 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:11.112053 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:11.112441 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:11.112446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:11.112450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:11.112641 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:11.113163 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:11.113170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:11.113533 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 01:03:11.113824 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 01:03:11.114892 (+ 1068us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":353,"threads_started":1}
I20260924 01:03:11.138733 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:11.123966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.8.254.67:38269)
0924 01:03:11.124731 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:11.124751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:11.124916 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:11.125252 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:11.125256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:11.125717 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:11.126001 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:11.126010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.127210 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.127215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:11.128840 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:11.128852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.130037 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.130041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:11.130262 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:11.131063 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:11.131081 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:11.131773 (+ 692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:11.135616 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:11.135622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:11.135626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:11.136184 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:11.136690 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:11.136694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:11.136700 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:11.136871 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:11.137557 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:11.137561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:11.137722 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 01:03:11.137845 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 01:03:11.138109 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":390,"threads_started":1}
I20260924 01:03:11.138767 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:11.124478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:38269 (local address 127.8.254.126:43107)
0924 01:03:11.124625 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:11.124629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:11.124840 (+ 211us) server_negotiation.cc:407] Connection header received
0924 01:03:11.125041 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:11.125046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:11.125121 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:11.125226 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:11.126147 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.127052 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.129048 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.129628 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.129747 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:11.131966 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:11.131975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:11.131982 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:11.132031 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:11.135478 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:11.136389 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:11.136397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:11.136403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:11.136573 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:11.137025 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:11.137031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:11.137035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:11.137417 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:11.137599 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:11.138013 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 01:03:11.138289 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 01:03:11.151439 17231 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:39957
I20260924 01:03:11.151461 17365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:39957 every 8 connection(s)
I20260924 01:03:11.152746 17231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:11.155031 17366 heartbeater.cc:347] Connected to a master server at 127.8.254.126:43107
I20260924 01:03:11.155267 17366 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:11.155920 17366 heartbeater.cc:511] Master 127.8.254.126:43107 requested a full tablet report, sending...
I20260924 01:03:11.157209 16881 ts_manager.cc:194] Registered new tserver with Master: c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:11.158077 16881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:51399
I20260924 01:03:11.159614 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 17231
I20260924 01:03:11.159812 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:03:11.160262 9209 external_mini_cluster.cc:1428] Setting key 9ad2a94efa7f68eb5fa3e94fd6a739b1
I20260924 01:03:11.163229 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 01:03:11 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 01:03:11 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:11 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:11.611290 17227 heartbeater.cc:503] Master 127.8.254.126:43107 was elected leader, sending a full tablet report...
I20260924 01:03:12.160913 17366 heartbeater.cc:503] Master 127.8.254.126:43107 was elected leader, sending a full tablet report...
W20260924 01:03:12.287243 17374 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.080s user 0.001s sys 0.014s
W20260924 01:03:12.288638 16976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:11.201735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.8.254.126:43107)
0924 01:03:11.201910 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:11.201915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:11.202176 (+ 261us) server_negotiation.cc:407] Connection header received
0924 01:03:11.202396 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:11.202400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:11.202466 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:11.202563 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:11.204001 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.204891 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.206096 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:11.206565 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:11.206661 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.288250 (+1081589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54894: BlockingRecv error: recv got EOF from 127.0.0.1:54894 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 01:03:12 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:12.306545 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.292609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.8.254.126:43107)
0924 01:03:12.292870 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.292876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.293119 (+ 243us) server_negotiation.cc:407] Connection header received
0924 01:03:12.293359 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.293365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.293455 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.293565 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:12.294899 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.295635 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.296912 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.297365 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.297508 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.301400 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:12.301414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:12.301418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:12.301453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:12.303894 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:12.304555 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:12.304561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:12.304565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:12.304692 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:12.305075 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:12.305081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:12.305084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:12.305362 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:12.305652 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.305976 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.306257 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 01:03:12.310675 16881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:12.317970 16881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:12.352912 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.330448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.8.254.66:43939)
0924 01:03:12.345073 (+ 14625us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.345083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.345110 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:03:12.345196 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.345206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.345447 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.345730 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.346410 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.347617 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.350368 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.351743 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.351884 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.352117 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.352435 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.352610 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14470,"thread_start_us":14330,"threads_started":1}
I20260924 01:03:12.352950 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.329992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43939 (local address 127.0.0.1:42006)
0924 01:03:12.330795 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.330843 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.331037 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.345683 (+ 14646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.345690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.345721 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.346182 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.346220 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.347770 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.347775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.350120 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.350133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.351955 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.351962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.352151 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.352204 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.352337 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.352555 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":28,"thread_start_us":378,"threads_started":1}
I20260924 01:03:12.354115 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.331212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40723 (local address 127.0.0.1:60954)
0924 01:03:12.342098 (+ 10886us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.342122 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.342281 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.345113 (+ 2832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.345121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.345170 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.345678 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.345695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.348219 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.348225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.351370 (+ 3145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.351383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.353311 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.353320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.353501 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.353538 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.353614 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.353748 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10725,"mutex_wait_us":155,"thread_start_us":8207,"threads_started":1}
I20260924 01:03:12.354264 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.331284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.8.254.67:39957)
0924 01:03:12.344618 (+ 13334us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.344628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.344654 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:03:12.344752 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.344759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.344978 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.345206 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.346225 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.347995 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.351482 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.352859 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.353014 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.353234 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.353800 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.353960 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13180,"thread_start_us":13039,"threads_started":1}
I20260924 01:03:12.356940 17157 tablet_service.cc:1467] Processing CreateTablet for tablet eb012968092e4a4a93892fe7cc69be58 (DEFAULT_TABLE table=test-table [id=1f7b35e289b04ccf9a50bfe971c13752]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:12.358100 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.331598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.8.254.65:40723)
0924 01:03:12.344297 (+ 12699us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.344310 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.344346 (+ 36us) server_negotiation.cc:407] Connection header received
0924 01:03:12.344467 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.344476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.344787 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.345029 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.346434 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.348046 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.351543 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.352883 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.353047 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.357698 (+ 4651us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.357771 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.357904 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12519,"thread_start_us":12366,"threads_started":1}
I20260924 01:03:12.358228 17157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb012968092e4a4a93892fe7cc69be58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.355144 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.330099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39957 (local address 127.0.0.1:34328)
0924 01:03:12.339737 (+ 9638us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.339763 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.339970 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.345396 (+ 5426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.345400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.345422 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.345941 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.345955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.348237 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.348241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.351310 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.351327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.353417 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.353425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.353610 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.353633 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.353743 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.353878 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9489,"mutex_wait_us":234,"thread_start_us":12289,"threads_started":1}
I20260924 01:03:12.361816 17018 tablet_service.cc:1467] Processing CreateTablet for tablet eb012968092e4a4a93892fe7cc69be58 (DEFAULT_TABLE table=test-table [id=1f7b35e289b04ccf9a50bfe971c13752]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:12.362411 17296 tablet_service.cc:1467] Processing CreateTablet for tablet eb012968092e4a4a93892fe7cc69be58 (DEFAULT_TABLE table=test-table [id=1f7b35e289b04ccf9a50bfe971c13752]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:12.363277 17018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb012968092e4a4a93892fe7cc69be58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.363668 17296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb012968092e4a4a93892fe7cc69be58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.370374 17391 tablet_bootstrap.cc:492] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Bootstrap starting.
I20260924 01:03:12.374310 17391 tablet_bootstrap.cc:654] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.375424 17391 log.cc:824] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:12.378158 17391 tablet_bootstrap.cc:492] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: No bootstrap required, opened a new log
I20260924 01:03:12.378594 17391 ts_tablet_manager.cc:1397] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 01:03:12.378595 17393 tablet_bootstrap.cc:492] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Bootstrap starting.
I20260924 01:03:12.379640 17394 tablet_bootstrap.cc:492] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Bootstrap starting.
I20260924 01:03:12.382216 17393 tablet_bootstrap.cc:654] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.382848 17394 tablet_bootstrap.cc:654] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.383522 17393 log.cc:824] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:12.383834 17394 log.cc:824] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:12.386634 17394 tablet_bootstrap.cc:492] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: No bootstrap required, opened a new log
I20260924 01:03:12.386646 17393 tablet_bootstrap.cc:492] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: No bootstrap required, opened a new log
I20260924 01:03:12.386905 17393 ts_tablet_manager.cc:1397] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 01:03:12.386942 17394 ts_tablet_manager.cc:1397] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 01:03:12.388129 17391 raft_consensus.cc:348] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.388533 17391 raft_consensus.cc:374] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.388660 17391 raft_consensus.cc:729] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce84215dc674449786297ef47b19a409, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.389360 17391 consensus_queue.cc:260] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.391130 17391 ts_tablet_manager.cc:1428] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260924 01:03:12.397604 17393 raft_consensus.cc:348] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.397605 17394 raft_consensus.cc:348] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.398036 17393 raft_consensus.cc:374] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.398036 17394 raft_consensus.cc:374] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.398144 17393 raft_consensus.cc:729] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc0179666914f528cb2911b53d87be1, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.398144 17394 raft_consensus.cc:729] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77662a1a990439eacb482b49021fa43, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.398772 17393 consensus_queue.cc:260] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.398772 17394 consensus_queue.cc:260] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.400616 17394 ts_tablet_manager.cc:1428] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 01:03:12.400784 17393 ts_tablet_manager.cc:1428] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260924 01:03:12.405123 17367 tablet.cc:2369] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:12.459448 17398 raft_consensus.cc:482] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:12.459816 17398 raft_consensus.cc:504] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.461531 17398 leader_election.cc:288] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce84215dc674449786297ef47b19a409 (127.8.254.66:43939), 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723)
I20260924 01:03:12.469684 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.462045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:37683 (local address 127.8.254.65:40723)
0924 01:03:12.462297 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.462304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.463606 (+ 1302us) server_negotiation.cc:407] Connection header received
0924 01:03:12.463752 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.463756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.463824 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.463916 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.464571 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.465595 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.467725 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.468746 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.468913 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.468992 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.469303 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.469433 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 01:03:12.469810 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.462503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40723 (local address 127.8.254.67:37683)
0924 01:03:12.463479 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.463504 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.463632 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.463944 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.463948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.463972 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.464377 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.464391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.465758 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.465764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.467562 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.467576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.468958 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.468964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.469127 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.469161 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.469255 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.469383 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":1,"thread_start_us":668,"threads_started":1}
I20260924 01:03:12.470737 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.461963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:34281 (local address 127.8.254.66:43939)
0924 01:03:12.462218 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.462247 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.463413 (+ 1166us) server_negotiation.cc:407] Connection header received
0924 01:03:12.463744 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.463750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.463824 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.463900 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.464688 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.465712 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.468263 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.469410 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.469557 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.469624 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.470371 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.470502 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:03:12.470865 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.462395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43939 (local address 127.8.254.67:34281)
0924 01:03:12.463286 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.463309 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.463534 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.464114 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.464117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.464156 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.464522 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.464534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.465878 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.465883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.468095 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.468109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.469812 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.469821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.470138 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.470230 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.470346 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.470484 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":1,"spinlock_wait_cycles":4480,"thread_start_us":468,"threads_started":1}
I20260924 01:03:12.470961 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb012968092e4a4a93892fe7cc69be58" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1" is_pre_election: true
I20260924 01:03:12.471563 17039 raft_consensus.cc:2454] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 0.
I20260924 01:03:12.472081 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb012968092e4a4a93892fe7cc69be58" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409" is_pre_election: true
I20260924 01:03:12.472277 17247 leader_election.cc:302] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1, c77662a1a990439eacb482b49021fa43; no voters:
I20260924 01:03:12.472666 17177 raft_consensus.cc:2454] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 0.
I20260924 01:03:12.472769 17398 raft_consensus.cc:2781] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:12.472916 17398 raft_consensus.cc:482] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:12.473028 17398 raft_consensus.cc:3037] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.475349 17398 raft_consensus.cc:504] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.475970 17398 leader_election.cc:288] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 election: Requested vote from peers ce84215dc674449786297ef47b19a409 (127.8.254.66:43939), 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723)
I20260924 01:03:12.476336 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb012968092e4a4a93892fe7cc69be58" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409"
I20260924 01:03:12.476478 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb012968092e4a4a93892fe7cc69be58" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1"
I20260924 01:03:12.476552 17177 raft_consensus.cc:3037] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.476696 17039 raft_consensus.cc:3037] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.479769 17177 raft_consensus.cc:2454] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 1.
I20260924 01:03:12.479789 17039 raft_consensus.cc:2454] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 1.
I20260924 01:03:12.480221 17250 leader_election.cc:302] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77662a1a990439eacb482b49021fa43, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:12.480566 17398 raft_consensus.cc:2781] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:12.481020 17398 raft_consensus.cc:686] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 1 LEADER]: Becoming Leader. State: Replica: c77662a1a990439eacb482b49021fa43, State: Running, Role: LEADER
I20260924 01:03:12.481478 17398 consensus_queue.cc:237] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } }
I20260924 01:03:12.485842 16881 catalog_manager.cc:5495] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 reported cstate change: term changed from 0 to 1, leader changed from <none> to c77662a1a990439eacb482b49021fa43 (127.8.254.67). New cstate: current_term: 1 leader_uuid: "c77662a1a990439eacb482b49021fa43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } health_report { overall_health: UNKNOWN } } }
I20260924 01:03:12.507426 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.502015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.8.254.67:39957)
0924 01:03:12.502252 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.502259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.502280 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:03:12.502537 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.502541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.502635 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.502789 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:12.503146 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.503930 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.505242 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.505645 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.505778 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.506280 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:12.506815 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:12.506981 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.507062 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.507145 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 01:03:12.515463 17039 raft_consensus.cc:1261] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Refusing update from remote peer c77662a1a990439eacb482b49021fa43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:12.515569 17177 raft_consensus.cc:1261] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Refusing update from remote peer c77662a1a990439eacb482b49021fa43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:12.516536 17398 consensus_queue.cc:1035] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:12.516840 17402 consensus_queue.cc:1035] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:12.526546 17405 mvcc.cc:205] Tried to move back new op lower bound from 7332707502132899840 to 7332707502008147968. Current Snapshot: MvccSnapshot[applied={T|T < 7332707502132899840}]
I20260924 01:03:12.528107 17406 mvcc.cc:205] Tried to move back new op lower bound from 7332707502132899840 to 7332707502008147968. Current Snapshot: MvccSnapshot[applied={T|T < 7332707502132899840}]
I20260924 01:03:12.540766 16880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:12.541131 16880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:12.543759 17408 mvcc.cc:205] Tried to move back new op lower bound from 7332707502132899840 to 7332707502008147968. Current Snapshot: MvccSnapshot[applied={T|T < 7332707502132899840}]
I20260924 01:03:12.546638 16880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Sending DeleteTablet for 3 replicas of tablet eb012968092e4a4a93892fe7cc69be58
I20260924 01:03:12.547729 17157 tablet_service.cc:1514] Processing DeleteTablet for tablet eb012968092e4a4a93892fe7cc69be58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:12 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:42006
I20260924 01:03:12.548740 17414 tablet_replica.cc:330] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: stopping tablet replica
I20260924 01:03:12.548648 17017 tablet_service.cc:1514] Processing DeleteTablet for tablet eb012968092e4a4a93892fe7cc69be58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:12 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60954
I20260924 01:03:12.548825 17018 tablet_service.cc:1514] Processing DeleteTablet for tablet eb012968092e4a4a93892fe7cc69be58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:12 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60954
I20260924 01:03:12.549239 17414 raft_consensus.cc:2229] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:12.549640 17415 tablet_replica.cc:330] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: stopping tablet replica
I20260924 01:03:12.549744 17414 raft_consensus.cc:2258] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:12.549724 17296 tablet_service.cc:1514] Processing DeleteTablet for tablet eb012968092e4a4a93892fe7cc69be58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:12 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:34328
I20260924 01:03:12.549998 17415 raft_consensus.cc:2229] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:12.550482 17415 raft_consensus.cc:2258] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:12.551350 16880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:12.552049 16880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:12.552398 17414 ts_tablet_manager.cc:1905] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:12.559659 17017 tablet_service.cc:1467] Processing CreateTablet for tablet 06ab4a481dc24920862afbf950b01d1b (DEFAULT_TABLE table=test-table [id=f7d53c606e564e34b65365a006f4892a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:12.559778 17157 tablet_service.cc:1467] Processing CreateTablet for tablet 06ab4a481dc24920862afbf950b01d1b (DEFAULT_TABLE table=test-table [id=f7d53c606e564e34b65365a006f4892a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:12.560170 17017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ab4a481dc24920862afbf950b01d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.560204 17157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ab4a481dc24920862afbf950b01d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.564374 17391 tablet_bootstrap.cc:492] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Bootstrap starting.
I20260924 01:03:12.564845 17415 ts_tablet_manager.cc:1905] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:12.565558 17295 tablet_service.cc:1467] Processing CreateTablet for tablet 06ab4a481dc24920862afbf950b01d1b (DEFAULT_TABLE table=test-table [id=f7d53c606e564e34b65365a006f4892a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:12.565972 17391 tablet_bootstrap.cc:654] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.566154 17295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ab4a481dc24920862afbf950b01d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.573612 17394 tablet_bootstrap.cc:492] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Bootstrap starting.
I20260924 01:03:12.574373 17415 ts_tablet_manager.cc:1918] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:12.574573 17415 log.cc:1196] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/eb012968092e4a4a93892fe7cc69be58
I20260924 01:03:12.574652 17393 tablet_bootstrap.cc:492] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Bootstrap starting.
I20260924 01:03:12.575112 17415 ts_tablet_manager.cc:1939] T eb012968092e4a4a93892fe7cc69be58 P 3cc0179666914f528cb2911b53d87be1: Deleting consensus metadata
I20260924 01:03:12.575475 17394 tablet_bootstrap.cc:654] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.576273 17393 tablet_bootstrap.cc:654] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.577183 16865 catalog_manager.cc:4841] TS 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723): tablet eb012968092e4a4a93892fe7cc69be58 (table test-table [id=1f7b35e289b04ccf9a50bfe971c13752]) successfully deleted
I20260924 01:03:12.578277 17414 ts_tablet_manager.cc:1918] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:12.578579 17414 log.cc:1196] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/eb012968092e4a4a93892fe7cc69be58
I20260924 01:03:12.579275 17414 ts_tablet_manager.cc:1939] T eb012968092e4a4a93892fe7cc69be58 P ce84215dc674449786297ef47b19a409: Deleting consensus metadata
I20260924 01:03:12.581336 16868 catalog_manager.cc:4841] TS ce84215dc674449786297ef47b19a409 (127.8.254.66:43939): tablet eb012968092e4a4a93892fe7cc69be58 (table test-table [id=1f7b35e289b04ccf9a50bfe971c13752]) successfully deleted
I20260924 01:03:12.583205 17391 tablet_bootstrap.cc:492] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: No bootstrap required, opened a new log
W20260924 01:03:12.583325 16865 catalog_manager.cc:4808] TS 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723): delete failed for tablet eb012968092e4a4a93892fe7cc69be58 because the tablet was not found. No further retry: Not found: Tablet not found: eb012968092e4a4a93892fe7cc69be58
I20260924 01:03:12.583704 17391 ts_tablet_manager.cc:1397] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20260924 01:03:12.585206 17394 tablet_bootstrap.cc:492] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: No bootstrap required, opened a new log
I20260924 01:03:12.585453 17394 ts_tablet_manager.cc:1397] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260924 01:03:12.586261 17393 tablet_bootstrap.cc:492] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: No bootstrap required, opened a new log
I20260924 01:03:12.586122 17391 raft_consensus.cc:348] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.586438 17391 raft_consensus.cc:374] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.586578 17393 ts_tablet_manager.cc:1397] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260924 01:03:12.586608 17391 raft_consensus.cc:729] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce84215dc674449786297ef47b19a409, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.586961 17416 tablet_replica.cc:330] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: stopping tablet replica
I20260924 01:03:12.586999 17391 consensus_queue.cc:260] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.587621 17391 ts_tablet_manager.cc:1428] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 01:03:12.588195 17416 raft_consensus.cc:2229] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:12.588306 17393 raft_consensus.cc:348] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.587621 17394 raft_consensus.cc:348] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.588536 17393 raft_consensus.cc:374] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.588640 17393 raft_consensus.cc:729] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc0179666914f528cb2911b53d87be1, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.588760 17394 raft_consensus.cc:374] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.589030 17394 raft_consensus.cc:729] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77662a1a990439eacb482b49021fa43, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.588912 17393 consensus_queue.cc:260] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.589416 17394 consensus_queue.cc:260] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.589576 17393 ts_tablet_manager.cc:1428] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 01:03:12.590310 17416 raft_consensus.cc:2258] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:12.590734 17394 ts_tablet_manager.cc:1428] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 01:03:12.592459 17416 ts_tablet_manager.cc:1905] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:12.600015 17416 ts_tablet_manager.cc:1918] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:12.600198 17416 log.cc:1196] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/eb012968092e4a4a93892fe7cc69be58
I20260924 01:03:12.600791 17416 ts_tablet_manager.cc:1939] T eb012968092e4a4a93892fe7cc69be58 P c77662a1a990439eacb482b49021fa43: Deleting consensus metadata
W20260924 01:03:12.601145 17089 tablet.cc:2369] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:12.602569 16866 catalog_manager.cc:4841] TS c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957): tablet eb012968092e4a4a93892fe7cc69be58 (table test-table [id=1f7b35e289b04ccf9a50bfe971c13752]) successfully deleted
W20260924 01:03:12.603483 17228 tablet.cc:2369] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:12.612354 17397 raft_consensus.cc:482] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:12.612671 17397 raft_consensus.cc:504] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.615362 17397 leader_election.cc:288] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:12.626394 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.616514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:37565 (local address 127.8.254.67:39957)
0924 01:03:12.616694 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.616700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.619307 (+ 2607us) server_negotiation.cc:407] Connection header received
0924 01:03:12.619601 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.619607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.619703 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.619776 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.620834 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.622018 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.624079 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.625246 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.625425 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.625502 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.625871 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.626027 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:03:12.626688 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.616682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:43491 (local address 127.8.254.65:40723)
0924 01:03:12.616842 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.616847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.619091 (+ 2244us) server_negotiation.cc:407] Connection header received
0924 01:03:12.619400 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.619406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.619482 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.619580 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.620619 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.621770 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.624429 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.625670 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.625862 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.625934 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.626268 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.626394 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:03:12.626808 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.615904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40723 (local address 127.8.254.66:43491)
0924 01:03:12.618934 (+ 3030us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.618958 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.619103 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.619936 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.619947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.619970 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.620437 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.620449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.621934 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.621942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.624265 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.624281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.625900 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.625906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.626076 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.626099 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.626207 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.626357 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2899,"mutex_wait_us":98,"thread_start_us":3178,"threads_started":1}
I20260924 01:03:12.627403 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ab4a481dc24920862afbf950b01d1b" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1" is_pre_election: true
I20260924 01:03:12.627676 17039 raft_consensus.cc:2454] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 0.
I20260924 01:03:12.627837 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.616346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39957 (local address 127.8.254.66:37565)
0924 01:03:12.619199 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.619218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.619433 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.620014 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.620019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.620063 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.620647 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.620661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.622142 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.622152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.623928 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.623941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.625470 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.625477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.625670 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.625709 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.627354 (+ 1645us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.627488 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2717,"mutex_wait_us":55,"thread_start_us":3382,"threads_started":1}
I20260924 01:03:12.628295 17108 leader_election.cc:302] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:12.628759 17397 raft_consensus.cc:2781] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:12.628885 17397 raft_consensus.cc:482] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:12.628971 17397 raft_consensus.cc:3037] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.628841 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ab4a481dc24920862afbf950b01d1b" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43" is_pre_election: true
I20260924 01:03:12.629359 17321 raft_consensus.cc:2454] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 0.
I20260924 01:03:12.631763 17397 raft_consensus.cc:504] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.632356 17397 leader_election.cc:288] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 election: Requested vote from peers 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:12.632774 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ab4a481dc24920862afbf950b01d1b" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1"
I20260924 01:03:12.633013 17039 raft_consensus.cc:3037] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.634450 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ab4a481dc24920862afbf950b01d1b" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43"
I20260924 01:03:12.634670 17321 raft_consensus.cc:3037] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.635367 17039 raft_consensus.cc:2454] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 1.
I20260924 01:03:12.635836 17108 leader_election.cc:302] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:12.636123 17397 raft_consensus.cc:2781] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:12.636656 17397 raft_consensus.cc:686] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 1 LEADER]: Becoming Leader. State: Replica: ce84215dc674449786297ef47b19a409, State: Running, Role: LEADER
I20260924 01:03:12.637265 17397 consensus_queue.cc:237] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.637933 17321 raft_consensus.cc:2454] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 1.
I20260924 01:03:12.644853 16880 catalog_manager.cc:5495] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce84215dc674449786297ef47b19a409 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } health_report { overall_health: UNKNOWN } } }
Sep 24 01:03:12 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211789, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.126@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:12.700996 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.684932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:43107 (local address 127.0.0.1:39042)
0924 01:03:12.685637 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.685663 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.685802 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.686175 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.686179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.686920 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:12.687185 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.687194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.688718 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.688722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.689959 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.689973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.691059 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.691067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.691276 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.692029 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:12.692051 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:12.694634 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:12.698440 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:12.698451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:12.698457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:12.698903 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:12.699326 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:12.699329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:12.699339 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:12.699478 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:12.700122 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:12.700135 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:12.700303 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.700461 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.700670 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":389,"threads_started":1}
I20260924 01:03:12.701247 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.684884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.8.254.126:43107)
0924 01:03:12.685125 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.685132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.685764 (+ 632us) server_negotiation.cc:407] Connection header received
0924 01:03:12.685919 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.685925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.686007 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.686131 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:12.687342 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.688586 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.690110 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.690817 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.690944 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.694839 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:12.694847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:12.694852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:12.694890 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:12.698297 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:12.699026 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:12.699031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:12.699035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:12.699174 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:12.699587 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:12.699593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:12.699598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:12.699960 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:12.700129 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.700583 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.700814 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 01:03:12.707079 16880 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:39042:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 01:03:12.708344 16880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:12.715651 17017 tablet_service.cc:1467] Processing CreateTablet for tablet 96791458995c4ad58987d9e471c55c77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9af2fa90f807496ea6a876689f494321]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:03:12.715993 17296 tablet_service.cc:1467] Processing CreateTablet for tablet 96791458995c4ad58987d9e471c55c77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9af2fa90f807496ea6a876689f494321]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:03:12.716274 17017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96791458995c4ad58987d9e471c55c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.716568 17296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96791458995c4ad58987d9e471c55c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.716452 17157 tablet_service.cc:1467] Processing CreateTablet for tablet 96791458995c4ad58987d9e471c55c77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9af2fa90f807496ea6a876689f494321]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 01:03:12.716964 17157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96791458995c4ad58987d9e471c55c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:12.724033 17394 tablet_bootstrap.cc:492] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: Bootstrap starting.
I20260924 01:03:12.724053 17393 tablet_bootstrap.cc:492] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: Bootstrap starting.
I20260924 01:03:12.725700 17393 tablet_bootstrap.cc:654] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.725775 17394 tablet_bootstrap.cc:654] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.726907 17391 tablet_bootstrap.cc:492] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: Bootstrap starting.
I20260924 01:03:12.727938 17393 tablet_bootstrap.cc:492] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: No bootstrap required, opened a new log
I20260924 01:03:12.728116 17393 ts_tablet_manager.cc:1397] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 01:03:12.728582 17391 tablet_bootstrap.cc:654] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:12.728724 17393 raft_consensus.cc:348] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.728955 17393 raft_consensus.cc:374] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.729058 17393 raft_consensus.cc:729] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc0179666914f528cb2911b53d87be1, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.729317 17393 consensus_queue.cc:260] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.729748 17399 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.730136 17399 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.731096 17391 tablet_bootstrap.cc:492] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: No bootstrap required, opened a new log
I20260924 01:03:12.731323 17391 ts_tablet_manager.cc:1397] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260924 01:03:12.733173 17391 raft_consensus.cc:348] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.733513 17391 raft_consensus.cc:374] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.733642 17391 raft_consensus.cc:729] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce84215dc674449786297ef47b19a409, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.734018 17393 ts_tablet_manager.cc:1428] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260924 01:03:12.733950 17391 consensus_queue.cc:260] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.734503 17397 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.734797 17397 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.735807 17391 ts_tablet_manager.cc:1428] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 01:03:12.742285 17394 tablet_bootstrap.cc:492] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: No bootstrap required, opened a new log
I20260924 01:03:12.742506 17394 ts_tablet_manager.cc:1397] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20260924 01:03:12.743259 17394 raft_consensus.cc:348] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.743535 17394 raft_consensus.cc:374] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:12.743631 17394 raft_consensus.cc:729] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77662a1a990439eacb482b49021fa43, State: Initialized, Role: FOLLOWER
I20260924 01:03:12.743886 17394 consensus_queue.cc:260] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.744540 17394 ts_tablet_manager.cc:1428] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 01:03:12.744434 17402 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.744776 17402 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.810475 17399 raft_consensus.cc:482] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:12.810714 17399 raft_consensus.cc:504] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.812306 17399 leader_election.cc:288] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce84215dc674449786297ef47b19a409 (127.8.254.66:43939), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:12.814116 17397 raft_consensus.cc:482] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:12.814369 17397 raft_consensus.cc:504] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.815197 17397 leader_election.cc:288] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:12.816046 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96791458995c4ad58987d9e471c55c77" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43" is_pre_election: true
I20260924 01:03:12.816349 17321 raft_consensus.cc:2454] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 0.
I20260924 01:03:12.816848 17109 leader_election.cc:302] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77662a1a990439eacb482b49021fa43, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:12.817510 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96791458995c4ad58987d9e471c55c77" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1" is_pre_election: true
I20260924 01:03:12.817847 17039 raft_consensus.cc:2454] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 0.
I20260924 01:03:12.817902 17397 raft_consensus.cc:2781] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:12.818015 17397 raft_consensus.cc:482] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:12.818143 17397 raft_consensus.cc:3037] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.821241 17397 raft_consensus.cc:504] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.822321 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96791458995c4ad58987d9e471c55c77" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1"
I20260924 01:03:12.822605 17039 raft_consensus.cc:3037] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.823225 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.812768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:53161 (local address 127.8.254.66:43939)
0924 01:03:12.813009 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.813018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.813953 (+ 935us) server_negotiation.cc:407] Connection header received
0924 01:03:12.814314 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.814322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.814436 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.814508 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.816131 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.817193 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.820331 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.821729 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.821885 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.821970 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.822797 (+ 827us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.822942 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:03:12.823588 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.813070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:43771 (local address 127.8.254.67:39957)
0924 01:03:12.813294 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.813300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.813668 (+ 368us) server_negotiation.cc:407] Connection header received
0924 01:03:12.813987 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.813995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.814079 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.814188 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:12.815486 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.816733 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.820723 (+ 3990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.822108 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.822322 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.822411 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.823098 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.823286 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 01:03:12.824426 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.812685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43939 (local address 127.8.254.65:53161)
0924 01:03:12.813846 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.813863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.814105 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.814758 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.814762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.814792 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.815877 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.815893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.817373 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.817378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.820102 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.820119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.822264 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.822272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.822528 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.822581 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.823749 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.823919 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":83,"thread_start_us":519,"threads_started":1}
I20260924 01:03:12.824493 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.812649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:39957 (local address 127.8.254.65:43771)
0924 01:03:12.813540 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.813562 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.813798 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.814499 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.814505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.814534 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:12.815156 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.815172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.817089 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.817094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.820530 (+ 3436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.820545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.822696 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.822703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.822871 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.822919 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.823876 (+ 957us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.824049 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":731,"thread_start_us":901,"threads_started":1}
I20260924 01:03:12.825289 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96791458995c4ad58987d9e471c55c77" candidate_uuid: "3cc0179666914f528cb2911b53d87be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409" is_pre_election: true
I20260924 01:03:12.825605 17177 raft_consensus.cc:2379] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cc0179666914f528cb2911b53d87be1 in current term 1: Already voted for candidate ce84215dc674449786297ef47b19a409 in this term.
I20260924 01:03:12.825711 17039 raft_consensus.cc:2454] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 1.
I20260924 01:03:12.825650 17321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96791458995c4ad58987d9e471c55c77" candidate_uuid: "ce84215dc674449786297ef47b19a409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43"
I20260924 01:03:12.825904 17321 raft_consensus.cc:3037] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:12.826109 17108 leader_election.cc:302] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:12.826434 17422 raft_consensus.cc:2781] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:12.826936 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96791458995c4ad58987d9e471c55c77" candidate_uuid: "3cc0179666914f528cb2911b53d87be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43" is_pre_election: true
I20260924 01:03:12.829566 17321 raft_consensus.cc:2454] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce84215dc674449786297ef47b19a409 in term 1.
I20260924 01:03:12.829656 16969 leader_election.cc:302] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1; no voters: c77662a1a990439eacb482b49021fa43, ce84215dc674449786297ef47b19a409
I20260924 01:03:12.829945 17397 leader_election.cc:288] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [CANDIDATE]: Term 1 election: Requested vote from peers 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:12.830561 17399 raft_consensus.cc:2726] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 01:03:12.830560 17422 raft_consensus.cc:686] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [term 1 LEADER]: Becoming Leader. State: Replica: ce84215dc674449786297ef47b19a409, State: Running, Role: LEADER
I20260924 01:03:12.830919 17422 consensus_queue.cc:237] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:12.832808 17397 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: TxnStatusTablet state changed. Reason: New leader ce84215dc674449786297ef47b19a409. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.833226 17397 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:03:12.834127 17437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 01:03:12.834414 17437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 01:03:12.834611 16879 catalog_manager.cc:5495] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce84215dc674449786297ef47b19a409 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } health_report { overall_health: UNKNOWN } } }
I20260924 01:03:12.872284 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.864523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.8.254.66:43939)
0924 01:03:12.864688 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.864693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.864791 (+ 98us) server_negotiation.cc:407] Connection header received
0924 01:03:12.865124 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.865129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.865236 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.865400 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:12.865980 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.867176 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.869233 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.869962 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.870108 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.870689 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:12.871531 (+ 842us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:12.871689 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.871820 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.871914 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 01:03:12.872357 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.864445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43939 (local address 127.0.0.1:42016)
0924 01:03:12.864688 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.864711 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.864892 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.865587 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.865593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.865606 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:03:12.865821 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.865833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.867324 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.867331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.869057 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.869075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.870143 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.870150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.870402 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.870481 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:03:12.871655 (+ 1174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:03:12.871682 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.871780 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.871941 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 01:03:12.877359 17039 raft_consensus.cc:1261] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Refusing update from remote peer ce84215dc674449786297ef47b19a409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:12.877748 17321 raft_consensus.cc:1261] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Refusing update from remote peer ce84215dc674449786297ef47b19a409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:12.878644 17422 consensus_queue.cc:1035] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:12.879374 17422 consensus_queue.cc:1035] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:12.891770 17402 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: TxnStatusTablet state changed. Reason: New leader ce84215dc674449786297ef47b19a409. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.892114 17402 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.894464 17435 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.894871 17435 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:03:12.895817 17402 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.896087 17402 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P c77662a1a990439eacb482b49021fa43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.897464 17435 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.897693 17435 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P ce84215dc674449786297ef47b19a409: This TxnStatusTablet replica's current role is: LEADER
I20260924 01:03:12.897576 17399 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: TxnStatusTablet state changed. Reason: New leader ce84215dc674449786297ef47b19a409. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.897882 17399 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.903482 17399 tablet_replica.cc:439] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce84215dc674449786297ef47b19a409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } } }
I20260924 01:03:12.903789 17399 tablet_replica.cc:442] T 96791458995c4ad58987d9e471c55c77 P 3cc0179666914f528cb2911b53d87be1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 01:03:12.915055 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.907636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.8.254.66:43939)
0924 01:03:12.907812 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.907820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.907837 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:03:12.907902 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.907907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.907992 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.908096 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:12.908702 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.909844 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.912114 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.912936 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.913070 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.913719 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:12.913884 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:12.913979 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.914583 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.914677 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:03:12.936326 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.928271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:57455 (local address 127.8.254.66:43939)
0924 01:03:12.928474 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:12.928480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:12.929287 (+ 807us) server_negotiation.cc:407] Connection header received
0924 01:03:12.929517 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:12.929523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:12.929614 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:12.929717 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:12.930396 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.931703 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.933903 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.934680 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.934825 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.935398 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:12.935557 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:12.935655 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:12.935834 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 01:03:12.935938 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":59}
I20260924 01:03:12.936781 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:12.928355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43939 (local address 127.8.254.66:57455)
0924 01:03:12.929137 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:12.929158 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:12.929356 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:12.929752 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:12.929756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:12.929769 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:03:12.930218 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.930234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.931859 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.931867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.933723 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:12.933741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:12.934898 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:12.934906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:12.935095 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:12.935179 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:03:12.935685 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:03:12.935697 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 01:03:12.935794 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 01:03:12.935955 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":23,"thread_start_us":433,"threads_started":1}
I20260924 01:03:12.948513 17038 raft_consensus.cc:1261] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Refusing update from remote peer ce84215dc674449786297ef47b19a409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:12.949441 17320 raft_consensus.cc:1261] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Refusing update from remote peer ce84215dc674449786297ef47b19a409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:12.949687 17422 consensus_queue.cc:1035] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:12.950381 17435 consensus_queue.cc:1035] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:13.048472 16879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:13.048774 16879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:13.052554 16879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Sending DeleteTablet for 3 replicas of tablet 06ab4a481dc24920862afbf950b01d1b
I20260924 01:03:13.053491 17017 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ab4a481dc24920862afbf950b01d1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60954
I20260924 01:03:13.053726 17156 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ab4a481dc24920862afbf950b01d1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:42006
I20260924 01:03:13.054136 17415 tablet_replica.cc:330] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: stopping tablet replica
I20260924 01:03:13.054368 17414 tablet_replica.cc:330] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: stopping tablet replica
I20260924 01:03:13.054581 17414 raft_consensus.cc:2229] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:13.055133 17414 raft_consensus.cc:2258] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.055482 17415 raft_consensus.cc:2229] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:13.055706 17415 raft_consensus.cc:2258] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.056496 17296 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ab4a481dc24920862afbf950b01d1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:34328
I20260924 01:03:13.056766 17416 tablet_replica.cc:330] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: stopping tablet replica
I20260924 01:03:13.056933 17415 ts_tablet_manager.cc:1905] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.056936 17416 raft_consensus.cc:2229] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:13.057126 17416 raft_consensus.cc:2258] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.058660 17416 ts_tablet_manager.cc:1905] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.063289 17414 ts_tablet_manager.cc:1905] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.064527 17415 ts_tablet_manager.cc:1918] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:03:13.064742 17415 log.cc:1196] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/06ab4a481dc24920862afbf950b01d1b
I20260924 01:03:13.065644 17415 ts_tablet_manager.cc:1939] T 06ab4a481dc24920862afbf950b01d1b P 3cc0179666914f528cb2911b53d87be1: Deleting consensus metadata
I20260924 01:03:13.066991 16865 catalog_manager.cc:4841] TS 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723): tablet 06ab4a481dc24920862afbf950b01d1b (table test-table [id=f7d53c606e564e34b65365a006f4892a]) successfully deleted
I20260924 01:03:13.074399 17416 ts_tablet_manager.cc:1918] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:03:13.074703 17416 log.cc:1196] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/06ab4a481dc24920862afbf950b01d1b
I20260924 01:03:13.075307 17416 ts_tablet_manager.cc:1939] T 06ab4a481dc24920862afbf950b01d1b P c77662a1a990439eacb482b49021fa43: Deleting consensus metadata
I20260924 01:03:13.076560 16866 catalog_manager.cc:4841] TS c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957): tablet 06ab4a481dc24920862afbf950b01d1b (table test-table [id=f7d53c606e564e34b65365a006f4892a]) successfully deleted
I20260924 01:03:13.080597 17414 ts_tablet_manager.cc:1918] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:03:13.080837 17414 log.cc:1196] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/06ab4a481dc24920862afbf950b01d1b
I20260924 01:03:13.081323 17414 ts_tablet_manager.cc:1939] T 06ab4a481dc24920862afbf950b01d1b P ce84215dc674449786297ef47b19a409: Deleting consensus metadata
I20260924 01:03:13.082538 16868 catalog_manager.cc:4841] TS ce84215dc674449786297ef47b19a409 (127.8.254.66:43939): tablet 06ab4a481dc24920862afbf950b01d1b (table test-table [id=f7d53c606e564e34b65365a006f4892a]) successfully deleted
Sep 24 01:03:13 dist-test-slave-5c7l krb5kdc[16836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 01:03:13 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:13 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 01:03:13.144100 16976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:13.128898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.8.254.126:43107)
0924 01:03:13.129054 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.129062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.129245 (+ 183us) server_negotiation.cc:407] Connection header received
0924 01:03:13.129493 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.129497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.129564 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.129673 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:13.131051 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.131791 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.133058 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.133494 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.133628 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.143828 (+ 10200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39058: BlockingRecv error: recv got EOF from 127.0.0.1:39058 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 01:03:13 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM
I20260924 01:03:13.162688 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.148044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.8.254.126:43107)
0924 01:03:13.148288 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.148297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.148816 (+ 519us) server_negotiation.cc:407] Connection header received
0924 01:03:13.149030 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.149035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.149123 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.149239 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:13.150588 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.151397 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.152644 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.153060 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.153202 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.157025 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:13.157030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:13.157034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:13.157070 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:13.159958 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:13.160614 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:13.160622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:13.160631 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:13.160752 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:13.161134 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:13.161140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:13.161143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:13.161429 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:13.161683 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:13.162015 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 01:03:13.162260 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 01:03:13.166426 16879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:13.167040 16879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:13.173606 17017 tablet_service.cc:1467] Processing CreateTablet for tablet f626fbe1f35a460fbd219e692c74c595 (DEFAULT_TABLE table=test-table [id=bcd241844d604d408f213f2e6a556780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:13.174070 17017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f626fbe1f35a460fbd219e692c74c595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:13.178398 17393 tablet_bootstrap.cc:492] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Bootstrap starting.
I20260924 01:03:13.178635 17156 tablet_service.cc:1467] Processing CreateTablet for tablet f626fbe1f35a460fbd219e692c74c595 (DEFAULT_TABLE table=test-table [id=bcd241844d604d408f213f2e6a556780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:13.179133 17156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f626fbe1f35a460fbd219e692c74c595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:13.179991 17393 tablet_bootstrap.cc:654] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:13.182714 17296 tablet_service.cc:1467] Processing CreateTablet for tablet f626fbe1f35a460fbd219e692c74c595 (DEFAULT_TABLE table=test-table [id=bcd241844d604d408f213f2e6a556780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:13.183243 17296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f626fbe1f35a460fbd219e692c74c595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:13.199179 17394 tablet_bootstrap.cc:492] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Bootstrap starting.
I20260924 01:03:13.200249 17391 tablet_bootstrap.cc:492] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Bootstrap starting.
I20260924 01:03:13.200587 17393 tablet_bootstrap.cc:492] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: No bootstrap required, opened a new log
I20260924 01:03:13.200800 17393 ts_tablet_manager.cc:1397] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.017s
I20260924 01:03:13.200877 17394 tablet_bootstrap.cc:654] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:13.201664 17393 raft_consensus.cc:348] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.201913 17393 raft_consensus.cc:374] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:13.202003 17393 raft_consensus.cc:729] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc0179666914f528cb2911b53d87be1, State: Initialized, Role: FOLLOWER
I20260924 01:03:13.202119 17391 tablet_bootstrap.cc:654] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:13.202247 17393 consensus_queue.cc:260] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.202857 17393 ts_tablet_manager.cc:1428] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:03:13.207903 17391 tablet_bootstrap.cc:492] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: No bootstrap required, opened a new log
I20260924 01:03:13.208107 17391 ts_tablet_manager.cc:1397] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 01:03:13.208985 17391 raft_consensus.cc:348] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.209321 17391 raft_consensus.cc:374] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:13.209463 17391 raft_consensus.cc:729] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce84215dc674449786297ef47b19a409, State: Initialized, Role: FOLLOWER
I20260924 01:03:13.209721 17391 consensus_queue.cc:260] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.210345 17391 ts_tablet_manager.cc:1428] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 01:03:13.211859 17394 tablet_bootstrap.cc:492] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: No bootstrap required, opened a new log
I20260924 01:03:13.212029 17394 ts_tablet_manager.cc:1397] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260924 01:03:13.212810 17394 raft_consensus.cc:348] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.213096 17394 raft_consensus.cc:374] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:13.213245 17394 raft_consensus.cc:729] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77662a1a990439eacb482b49021fa43, State: Initialized, Role: FOLLOWER
I20260924 01:03:13.213537 17394 consensus_queue.cc:260] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.214792 17394 ts_tablet_manager.cc:1428] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 01:03:13.480458 17402 raft_consensus.cc:482] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:13.480713 17402 raft_consensus.cc:504] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.481429 17402 leader_election.cc:288] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723), ce84215dc674449786297ef47b19a409 (127.8.254.66:43939)
I20260924 01:03:13.482043 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f626fbe1f35a460fbd219e692c74c595" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1" is_pre_election: true
I20260924 01:03:13.482142 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f626fbe1f35a460fbd219e692c74c595" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409" is_pre_election: true
I20260924 01:03:13.482328 17039 raft_consensus.cc:2454] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 0.
I20260924 01:03:13.482419 17177 raft_consensus.cc:2454] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 0.
I20260924 01:03:13.482913 17250 leader_election.cc:302] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77662a1a990439eacb482b49021fa43, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:13.483255 17402 raft_consensus.cc:2781] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:13.483368 17402 raft_consensus.cc:482] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:13.483476 17402 raft_consensus.cc:3037] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:13.485463 17402 raft_consensus.cc:504] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.486183 17402 leader_election.cc:288] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 election: Requested vote from peers 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723), ce84215dc674449786297ef47b19a409 (127.8.254.66:43939)
I20260924 01:03:13.486938 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f626fbe1f35a460fbd219e692c74c595" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cc0179666914f528cb2911b53d87be1"
I20260924 01:03:13.487370 17039 raft_consensus.cc:3037] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:13.494365 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f626fbe1f35a460fbd219e692c74c595" candidate_uuid: "c77662a1a990439eacb482b49021fa43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409"
I20260924 01:03:13.494624 17177 raft_consensus.cc:3037] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:13.497283 17039 raft_consensus.cc:2454] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 1.
I20260924 01:03:13.497689 17177 raft_consensus.cc:2454] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77662a1a990439eacb482b49021fa43 in term 1.
I20260924 01:03:13.498366 17250 leader_election.cc:302] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77662a1a990439eacb482b49021fa43, ce84215dc674449786297ef47b19a409; no voters:
I20260924 01:03:13.498808 17402 raft_consensus.cc:2781] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:13.500566 17402 raft_consensus.cc:686] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 1 LEADER]: Becoming Leader. State: Replica: c77662a1a990439eacb482b49021fa43, State: Running, Role: LEADER
I20260924 01:03:13.500939 17402 consensus_queue.cc:237] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.504215 16879 catalog_manager.cc:5495] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 reported cstate change: term changed from 0 to 1, leader changed from <none> to c77662a1a990439eacb482b49021fa43 (127.8.254.67). New cstate: current_term: 1 leader_uuid: "c77662a1a990439eacb482b49021fa43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } health_report { overall_health: HEALTHY } } }
I20260924 01:03:13.583290 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.578354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.8.254.67:39957)
0924 01:03:13.578828 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.578833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.578848 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:13.578913 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.578916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.578977 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.579088 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:13.579481 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.580169 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.581509 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.581925 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.582065 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.582544 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:13.582645 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:13.582748 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:13.582916 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 01:03:13.582997 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260924 01:03:13.586462 17177 raft_consensus.cc:1261] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Refusing update from remote peer c77662a1a990439eacb482b49021fa43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:13.586462 17039 raft_consensus.cc:1261] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Refusing update from remote peer c77662a1a990439eacb482b49021fa43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:13.587208 17464 consensus_queue.cc:1035] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:13.587726 17402 consensus_queue.cc:1035] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:13.607344 16881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:13.607589 16881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:13.611160 16881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Sending DeleteTablet for 3 replicas of tablet f626fbe1f35a460fbd219e692c74c595
I20260924 01:03:13.611981 17156 tablet_service.cc:1514] Processing DeleteTablet for tablet f626fbe1f35a460fbd219e692c74c595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:42006
I20260924 01:03:13.612879 17017 tablet_service.cc:1514] Processing DeleteTablet for tablet f626fbe1f35a460fbd219e692c74c595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60954
I20260924 01:03:13.618651 17476 tablet_replica.cc:330] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: stopping tablet replica
I20260924 01:03:13.620563 17296 tablet_service.cc:1514] Processing DeleteTablet for tablet f626fbe1f35a460fbd219e692c74c595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:34328
I20260924 01:03:13.625878 16881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:13.626762 16881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:13.635812 17295 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ccbc7e5b147a1a146696fc7e0da94 (DEFAULT_TABLE table=test-table [id=413da0ad477b4cea9164be904815e30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:13.636247 17295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1ccbc7e5b147a1a146696fc7e0da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:13.639519 17018 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ccbc7e5b147a1a146696fc7e0da94 (DEFAULT_TABLE table=test-table [id=413da0ad477b4cea9164be904815e30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:13.639976 17018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1ccbc7e5b147a1a146696fc7e0da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:13.643006 17156 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ccbc7e5b147a1a146696fc7e0da94 (DEFAULT_TABLE table=test-table [id=413da0ad477b4cea9164be904815e30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:13.643445 17156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1ccbc7e5b147a1a146696fc7e0da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:13.649641 17391 tablet_bootstrap.cc:492] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Bootstrap starting.
I20260924 01:03:13.651450 17391 tablet_bootstrap.cc:654] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:13.671165 17393 tablet_bootstrap.cc:492] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Bootstrap starting.
I20260924 01:03:13.673539 17393 tablet_bootstrap.cc:654] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:13.694971 17394 tablet_bootstrap.cc:492] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Bootstrap starting.
I20260924 01:03:13.697396 17394 tablet_bootstrap.cc:654] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:13.719635 17478 tablet_replica.cc:330] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: stopping tablet replica
I20260924 01:03:13.720059 17478 raft_consensus.cc:2229] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:13.720741 17478 raft_consensus.cc:2258] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.723863 17393 tablet_bootstrap.cc:492] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: No bootstrap required, opened a new log
I20260924 01:03:13.724958 17393 ts_tablet_manager.cc:1397] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.012s
I20260924 01:03:13.726151 17393 raft_consensus.cc:348] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.729194 17394 tablet_bootstrap.cc:492] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: No bootstrap required, opened a new log
I20260924 01:03:13.729777 17476 raft_consensus.cc:2229] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:13.730510 17476 raft_consensus.cc:2258] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.732084 17394 ts_tablet_manager.cc:1397] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.009s
I20260924 01:03:13.732801 17394 raft_consensus.cc:348] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.733029 17394 raft_consensus.cc:374] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:13.733134 17394 raft_consensus.cc:729] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77662a1a990439eacb482b49021fa43, State: Initialized, Role: FOLLOWER
I20260924 01:03:13.733392 17394 consensus_queue.cc:260] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.734112 17478 ts_tablet_manager.cc:1905] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.727843 17477 tablet_replica.cc:330] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: stopping tablet replica
I20260924 01:03:13.738627 17393 raft_consensus.cc:374] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:13.738740 17393 raft_consensus.cc:729] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc0179666914f528cb2911b53d87be1, State: Initialized, Role: FOLLOWER
I20260924 01:03:13.738930 17477 raft_consensus.cc:2229] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:13.739240 17477 raft_consensus.cc:2258] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.741021 17476 ts_tablet_manager.cc:1905] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.741673 17394 ts_tablet_manager.cc:1428] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260924 01:03:13.742031 17393 consensus_queue.cc:260] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.744971 17399 raft_consensus.cc:482] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:13.745105 17399 raft_consensus.cc:504] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.746058 17399 leader_election.cc:288] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce84215dc674449786297ef47b19a409 (127.8.254.66:43939), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:13.746837 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ccbc7e5b147a1a146696fc7e0da94" candidate_uuid: "3cc0179666914f528cb2911b53d87be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43" is_pre_election: true
I20260924 01:03:13.746869 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ccbc7e5b147a1a146696fc7e0da94" candidate_uuid: "3cc0179666914f528cb2911b53d87be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409" is_pre_election: true
I20260924 01:03:13.747079 17320 raft_consensus.cc:2454] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cc0179666914f528cb2911b53d87be1 in term 0.
I20260924 01:03:13.747684 16969 leader_election.cc:302] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1, c77662a1a990439eacb482b49021fa43; no voters:
I20260924 01:03:13.747962 17399 raft_consensus.cc:2781] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:13.748086 17399 raft_consensus.cc:482] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 01:03:13.748152 16971 leader_election.cc:341] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ce84215dc674449786297ef47b19a409 (127.8.254.66:43939): Illegal state: must be running to vote when last-logged opid is not known
I20260924 01:03:13.748204 17399 raft_consensus.cc:3037] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:13.748811 17393 ts_tablet_manager.cc:1428] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20260924 01:03:13.750917 17478 ts_tablet_manager.cc:1918] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:13.751085 17478 log.cc:1196] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/f626fbe1f35a460fbd219e692c74c595
I20260924 01:03:13.751555 17478 ts_tablet_manager.cc:1939] T f626fbe1f35a460fbd219e692c74c595 P c77662a1a990439eacb482b49021fa43: Deleting consensus metadata
I20260924 01:03:13.752794 16866 catalog_manager.cc:4841] TS c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957): tablet f626fbe1f35a460fbd219e692c74c595 (table test-table [id=bcd241844d604d408f213f2e6a556780]) successfully deleted
I20260924 01:03:13.753574 17391 tablet_bootstrap.cc:492] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: No bootstrap required, opened a new log
I20260924 01:03:13.753616 17399 raft_consensus.cc:504] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.753921 17391 ts_tablet_manager.cc:1397] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Time spent bootstrapping tablet: real 0.104s user 0.050s sys 0.048s
I20260924 01:03:13.754554 17477 ts_tablet_manager.cc:1905] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.755029 17399 leader_election.cc:288] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 election: Requested vote from peers ce84215dc674449786297ef47b19a409 (127.8.254.66:43939), c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957)
I20260924 01:03:13.758783 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ccbc7e5b147a1a146696fc7e0da94" candidate_uuid: "3cc0179666914f528cb2911b53d87be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce84215dc674449786297ef47b19a409"
I20260924 01:03:13.759183 17391 raft_consensus.cc:348] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
W20260924 01:03:13.759702 16971 leader_election.cc:341] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ce84215dc674449786297ef47b19a409 (127.8.254.66:43939): Illegal state: must be running to vote when last-logged opid is not known
I20260924 01:03:13.759816 17477 ts_tablet_manager.cc:1918] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:13.759946 17477 log.cc:1196] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/f626fbe1f35a460fbd219e692c74c595
I20260924 01:03:13.759997 17476 ts_tablet_manager.cc:1918] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:13.760192 17476 log.cc:1196] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/f626fbe1f35a460fbd219e692c74c595
I20260924 01:03:13.760614 17477 ts_tablet_manager.cc:1939] T f626fbe1f35a460fbd219e692c74c595 P 3cc0179666914f528cb2911b53d87be1: Deleting consensus metadata
I20260924 01:03:13.760865 17476 ts_tablet_manager.cc:1939] T f626fbe1f35a460fbd219e692c74c595 P ce84215dc674449786297ef47b19a409: Deleting consensus metadata
I20260924 01:03:13.761941 17391 raft_consensus.cc:374] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:13.762127 17391 raft_consensus.cc:729] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce84215dc674449786297ef47b19a409, State: Initialized, Role: FOLLOWER
I20260924 01:03:13.762149 16868 catalog_manager.cc:4841] TS ce84215dc674449786297ef47b19a409 (127.8.254.66:43939): tablet f626fbe1f35a460fbd219e692c74c595 (table test-table [id=bcd241844d604d408f213f2e6a556780]) successfully deleted
I20260924 01:03:13.762535 17391 consensus_queue.cc:260] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.762856 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ccbc7e5b147a1a146696fc7e0da94" candidate_uuid: "3cc0179666914f528cb2911b53d87be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77662a1a990439eacb482b49021fa43"
I20260924 01:03:13.763105 17320 raft_consensus.cc:3037] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:13.763981 16865 catalog_manager.cc:4841] TS 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723): tablet f626fbe1f35a460fbd219e692c74c595 (table test-table [id=bcd241844d604d408f213f2e6a556780]) successfully deleted
I20260924 01:03:13.765707 17391 ts_tablet_manager.cc:1428] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20260924 01:03:13.766040 17320 raft_consensus.cc:2454] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cc0179666914f528cb2911b53d87be1 in term 1.
I20260924 01:03:13.767033 16969 leader_election.cc:302] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cc0179666914f528cb2911b53d87be1, c77662a1a990439eacb482b49021fa43; no voters: ce84215dc674449786297ef47b19a409
I20260924 01:03:13.767897 17482 raft_consensus.cc:2781] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:13.768201 17482 raft_consensus.cc:686] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc0179666914f528cb2911b53d87be1, State: Running, Role: LEADER
I20260924 01:03:13.768661 17482 consensus_queue.cc:237] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } }
I20260924 01:03:13.772421 16879 catalog_manager.cc:5495] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cc0179666914f528cb2911b53d87be1 (127.8.254.65). New cstate: current_term: 1 leader_uuid: "3cc0179666914f528cb2911b53d87be1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cc0179666914f528cb2911b53d87be1" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 40723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 } health_report { overall_health: UNKNOWN } } }
I20260924 01:03:13.810062 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.803381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.8.254.65:40723)
0924 01:03:13.804368 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.804375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.804407 (+ 32us) server_negotiation.cc:407] Connection header received
0924 01:03:13.804535 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.804540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.804650 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.804826 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:13.805198 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.806335 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.807583 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.808202 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.808324 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.808717 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:13.809455 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:13.809596 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:13.809641 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 01:03:13.809718 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":647,"threads_started":1}
I20260924 01:03:13.827689 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.820889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:46193 (local address 127.8.254.66:43939)
0924 01:03:13.821606 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.821612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.821643 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:13.821844 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.821850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.821944 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.822061 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:13.822637 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.823715 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.825714 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.826253 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.826365 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.826915 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:13.827049 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:13.827125 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:13.827315 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 01:03:13.827384 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":369,"threads_started":1}
I20260924 01:03:13.827842 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.820812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:43939 (local address 127.8.254.65:46193)
0924 01:03:13.821569 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:13.821585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:13.821700 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:13.822076 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:13.822079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:13.822089 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:03:13.822437 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:13.822447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.823885 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.823890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:13.825537 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:13.825553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.826425 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.826434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:13.826618 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.826705 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:03:13.827162 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:03:13.827176 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 01:03:13.827270 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 01:03:13.827441 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":420,"threads_started":1}
I20260924 01:03:13.837399 17176 raft_consensus.cc:3037] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:13.837527 17320 raft_consensus.cc:1261] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Refusing update from remote peer 3cc0179666914f528cb2911b53d87be1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:13.838258 17482 consensus_queue.cc:1035] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77662a1a990439eacb482b49021fa43" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:13.841107 17176 raft_consensus.cc:1261] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Refusing update from remote peer 3cc0179666914f528cb2911b53d87be1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:13.841967 17482 consensus_queue.cc:1035] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce84215dc674449786297ef47b19a409" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:13.871738 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.865834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:49819 (local address 127.8.254.65:40723)
0924 01:03:13.866021 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.866027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.866717 (+ 690us) server_negotiation.cc:407] Connection header received
0924 01:03:13.866926 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.866932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.867018 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.867116 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:13.867746 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.868541 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.869874 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.870316 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.870412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.870932 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:13.871044 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:13.871113 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:13.871298 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 01:03:13.871379 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 01:03:13.871809 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.865852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:40723 (local address 127.8.254.66:49819)
0924 01:03:13.866606 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:13.866628 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:13.866777 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:13.867152 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:13.867156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:13.867168 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 01:03:13.867565 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:13.867575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.868667 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.868671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:13.869729 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:13.869740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.870495 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.870501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:13.870696 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.870753 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 01:03:13.871145 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 01:03:13.871157 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 01:03:13.871248 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 01:03:13.871372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":769,"threads_started":1}
I20260924 01:03:13.901995 16881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:13.902303 16881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:13.905520 16881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1929b03b7a4744bc9c74f392644e5797: Sending DeleteTablet for 3 replicas of tablet 0f1ccbc7e5b147a1a146696fc7e0da94
I20260924 01:03:13.906384 17017 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ccbc7e5b147a1a146696fc7e0da94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60954
I20260924 01:03:13.906526 17156 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ccbc7e5b147a1a146696fc7e0da94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:42006
I20260924 01:03:13.906723 17477 tablet_replica.cc:330] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: stopping tablet replica
I20260924 01:03:13.906705 17296 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ccbc7e5b147a1a146696fc7e0da94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:13 UTC) from {username='oryx', principal='oryx/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:34328
I20260924 01:03:13.906816 17476 tablet_replica.cc:330] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: stopping tablet replica
I20260924 01:03:13.906939 17477 raft_consensus.cc:2229] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:13.906982 17478 tablet_replica.cc:330] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: stopping tablet replica
I20260924 01:03:13.906994 17476 raft_consensus.cc:2229] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:13.907173 17478 raft_consensus.cc:2229] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:13.907218 17476 raft_consensus.cc:2258] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.907351 17478 raft_consensus.cc:2258] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.907903 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 16952
I20260924 01:03:13.909118 17478 ts_tablet_manager.cc:1905] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.914726 17477 raft_consensus.cc:2258] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:13.915125 17476 ts_tablet_manager.cc:1905] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.916074 17477 ts_tablet_manager.cc:1905] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:13.917258 17478 ts_tablet_manager.cc:1918] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:03:13.917397 17478 log.cc:1196] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/0f1ccbc7e5b147a1a146696fc7e0da94
I20260924 01:03:13.917768 17478 ts_tablet_manager.cc:1939] T 0f1ccbc7e5b147a1a146696fc7e0da94 P c77662a1a990439eacb482b49021fa43: Deleting consensus metadata
I20260924 01:03:13.918812 16866 catalog_manager.cc:4841] TS c77662a1a990439eacb482b49021fa43 (127.8.254.67:39957): tablet 0f1ccbc7e5b147a1a146696fc7e0da94 (table test-table [id=413da0ad477b4cea9164be904815e30f]) successfully deleted
Sep 24 01:03:13 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.65@KRBTEST.COM
I20260924 01:03:13.922542 17476 ts_tablet_manager.cc:1918] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:03:13.922690 17476 log.cc:1196] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/0f1ccbc7e5b147a1a146696fc7e0da94
I20260924 01:03:13.923199 17476 ts_tablet_manager.cc:1939] T 0f1ccbc7e5b147a1a146696fc7e0da94 P ce84215dc674449786297ef47b19a409: Deleting consensus metadata
I20260924 01:03:13.923518 17477 ts_tablet_manager.cc:1918] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 01:03:13.923746 17477 log.cc:1196] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/0f1ccbc7e5b147a1a146696fc7e0da94
I20260924 01:03:13.924307 17477 ts_tablet_manager.cc:1939] T 0f1ccbc7e5b147a1a146696fc7e0da94 P 3cc0179666914f528cb2911b53d87be1: Deleting consensus metadata
I20260924 01:03:13.924573 16868 catalog_manager.cc:4841] TS ce84215dc674449786297ef47b19a409 (127.8.254.66:43939): tablet 0f1ccbc7e5b147a1a146696fc7e0da94 (table test-table [id=413da0ad477b4cea9164be904815e30f]) successfully deleted
I20260924 01:03:13.925865 16865 catalog_manager.cc:4841] TS 3cc0179666914f528cb2911b53d87be1 (127.8.254.65:40723): tablet 0f1ccbc7e5b147a1a146696fc7e0da94 (table test-table [id=413da0ad477b4cea9164be904815e30f]) successfully deleted
I20260924 01:03:13.930576 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:13.909248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.8.254.65:40723)
0924 01:03:13.909427 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:13.909432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:13.910799 (+ 1367us) server_negotiation.cc:407] Connection header received
0924 01:03:13.911019 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:13.911026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:13.912651 (+ 1625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:13.912770 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:13.914433 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.915634 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.917485 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:13.918145 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:13.918275 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:13.924064 (+ 5789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:13.924080 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:13.924099 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:13.924139 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:13.927656 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:13.928358 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:13.928375 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:13.928378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:13.928512 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:13.928921 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:13.928930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:13.928935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:13.929310 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:13.929635 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:13.929950 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 01:03:13.930183 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 01:03:13.931273 17083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:14.037914 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 16952
I20260924 01:03:14.057509 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 17092
Sep 24 01:03:14 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.66@KRBTEST.COM
I20260924 01:03:14.075670 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:14.058489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.8.254.66:43939)
0924 01:03:14.058690 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:14.058697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:14.058716 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:03:14.058783 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:14.058787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:14.058849 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:14.058973 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:14.060600 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:14.061665 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:14.063049 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:14.063561 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:14.063696 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:14.069528 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:14.069545 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:14.069562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:14.069601 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:14.072681 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:14.073399 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:14.073407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:14.073409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:14.073546 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:14.073960 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:14.073965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:14.073967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:14.074393 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:14.074715 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:14.075071 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 01:03:14.075337 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:03:14.076471 17222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:14.230322 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 17092
I20260924 01:03:14.256788 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 17231
Sep 24 01:03:14 dist-test-slave-5c7l krb5kdc[16836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.254.67@KRBTEST.COM
I20260924 01:03:14.272015 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:14.257567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.8.254.67:39957)
0924 01:03:14.258056 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:14.258061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:14.258076 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:14.258130 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:14.258134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:14.258257 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:14.258374 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:14.259668 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:14.260378 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:14.261644 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:14.262086 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:14.262250 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:14.266565 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:14.266581 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:14.266597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:14.266633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:14.269458 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:14.270191 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:14.270251 (+ 60us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:14.270254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:14.270380 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:14.270763 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:14.270766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:14.270768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:14.271114 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:14.271360 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:14.271595 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 01:03:14.271770 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
I20260924 01:03:14.272663 17361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:14.397899 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 17231
I20260924 01:03:14.420913 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 16849
I20260924 01:03:14.422124 16921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:14.539793 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 16849
2026-09-24T01:03:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6857 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:03:14 dist-test-slave-5c7l krb5kdc[17507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:03:14 dist-test-slave-5c7l krb5kdc[17507](info): set up 2 sockets
Sep 24 01:03:14 dist-test-slave-5c7l krb5kdc[17507](info): commencing operation
krb5kdc: starting...
W20260924 01:03:15.640350 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:03:15 dist-test-slave-5c7l krb5kdc[17507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:03:15.866641 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:34869
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:34869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:16.192677 17520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:16.193006 17520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:16.193172 17520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:16.200289 17520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:16.200433 17520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:16.200546 17520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:16.200642 17520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:16.200734 17520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:16.210507 17520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:34869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:34869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:16.211814 17520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:16.213083 17520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:16.225028 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:16.225085 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:16.225001 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:16.225852 17520 server_base.cc:1034] running on GCE node
I20260924 01:03:16.226769 17520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:16.228230 17520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:16.229485 17520 hybrid_clock.cc:648] HybridClock initialized: now 1790211796229426 us; error 49 us; skew 500 ppm
I20260924 01:03:16.229846 17520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T01:03:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1763 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:03:16 dist-test-slave-5c7l krb5kdc[17534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:03:16 dist-test-slave-5c7l krb5kdc[17534](info): set up 2 sockets
Sep 24 01:03:16 dist-test-slave-5c7l krb5kdc[17534](info): commencing operation
krb5kdc: starting...
W20260924 01:03:17.412176 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:03:17 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:03:17.649969 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44031
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:44031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:17.976078 17547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:17.976388 17547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:17.976480 17547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:17.983745 17547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:17.983860 17547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:17.983906 17547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:17.983955 17547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:17.983999 17547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:17.993928 17547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:44031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:17.995327 17547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:17.996634 17547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:18.008877 17552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:18.009093 17555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:18.009115 17553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:18.010105 17547 server_base.cc:1034] running on GCE node
I20260924 01:03:18.011322 17547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:18.012796 17547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:18.014074 17547 hybrid_clock.cc:648] HybridClock initialized: now 1790211798014003 us; error 55 us; skew 500 ppm
I20260924 01:03:18.014496 17547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:18 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:18.017820 17547 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:03:18.019377 17547 webserver.cc:460] Webserver started at http://127.8.254.126:45827/ using document root <none> and password file <none>
I20260924 01:03:18.020100 17547 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:18.020248 17547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:18.020582 17547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:18.023741 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "c908fb4b3dba4f228070043b7c62169e"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "d75e38cd02fde52226675ac08e6ded52"
server_key_iv: "07a329f1d4dc7c13896ef9b63fc16fb5"
server_key_version: "encryptionkey@0"
I20260924 01:03:18.024556 17547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "c908fb4b3dba4f228070043b7c62169e"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "d75e38cd02fde52226675ac08e6ded52"
server_key_iv: "07a329f1d4dc7c13896ef9b63fc16fb5"
server_key_version: "encryptionkey@0"
I20260924 01:03:18.029846 17547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 01:03:18.033493 17562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:18.034942 17547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:18.035131 17547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "c908fb4b3dba4f228070043b7c62169e"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "d75e38cd02fde52226675ac08e6ded52"
server_key_iv: "07a329f1d4dc7c13896ef9b63fc16fb5"
server_key_version: "encryptionkey@0"
I20260924 01:03:18.035344 17547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:18.047093 17547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:18.051796 17547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:18.052090 17547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:18.070850 17547 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:44031
I20260924 01:03:18.070874 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:44031 every 8 connection(s)
I20260924 01:03:18.072320 17547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:03:18.076233 17614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:18.082582 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 17547
I20260924 01:03:18.082809 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:03:18.083441 9209 external_mini_cluster.cc:1428] Setting key fd7412e728d7cf080c4d70eaa447c778
I20260924 01:03:18.087719 17614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Bootstrap starting.
I20260924 01:03:18.091471 17614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:18.092813 17614 log.cc:824] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:18.095990 17614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: No bootstrap required, opened a new log
Sep 24 01:03:18 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:18.105890 17614 raft_consensus.cc:348] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:18.106474 17614 raft_consensus.cc:374] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:18.106585 17614 raft_consensus.cc:729] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c908fb4b3dba4f228070043b7c62169e, State: Initialized, Role: FOLLOWER
I20260924 01:03:18.107334 17614 consensus_queue.cc:260] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:18.107688 17614 raft_consensus.cc:388] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:03:18.107844 17614 raft_consensus.cc:482] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:03:18.108027 17614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:18.109454 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:18.085498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.8.254.126:44031)
0924 01:03:18.086341 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:18.086359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:18.086410 (+ 51us) server_negotiation.cc:407] Connection header received
0924 01:03:18.088959 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:18.088986 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:18.089505 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:18.089975 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:18.091520 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:18.093583 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:18.095197 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:18.096211 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:18.096392 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:18.100213 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:18.100230 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:18.100251 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:18.100300 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:18.104541 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:18.105326 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:18.105334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:18.105346 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:18.105561 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:18.105928 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:18.105933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:18.105936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:18.106702 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:18.106987 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:18.107446 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 01:03:18.107953 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":352,"threads_started":1}
I20260924 01:03:18.110961 17614 raft_consensus.cc:504] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:18.111474 17614 leader_election.cc:302] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c908fb4b3dba4f228070043b7c62169e; no voters:
I20260924 01:03:18.112082 17614 leader_election.cc:288] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:03:18.112236 17619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:18.113328 17619 raft_consensus.cc:686] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 LEADER]: Becoming Leader. State: Replica: c908fb4b3dba4f228070043b7c62169e, State: Running, Role: LEADER
I20260924 01:03:18.113830 17619 consensus_queue.cc:237] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:18.114302 17614 sys_catalog.cc:564] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:18.116655 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c908fb4b3dba4f228070043b7c62169e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } } }
I20260924 01:03:18.117120 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:18.116931 17621 sys_catalog.cc:455] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c908fb4b3dba4f228070043b7c62169e. Latest consensus state: current_term: 1 leader_uuid: "c908fb4b3dba4f228070043b7c62169e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } } }
I20260924 01:03:18.117244 17621 sys_catalog.cc:458] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:18.124790 17627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:18.130445 17627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:18.140580 17627 catalog_manager.cc:1347] Generated new cluster ID: ee7aae3f84c54cdd9e234d2c6e1927e7
I20260924 01:03:18.140713 17627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:18.157333 17627 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:03:18.158793 17627 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:18.169104 17627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Generated new TSK 0
I20260924 01:03:18.170038 17627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:03:18.238529 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44031
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:18.573927 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:18.574338 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:18.574522 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:18.581779 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:18.581894 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:18.582084 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:18.592245 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:18.593662 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:18.595026 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:18.608117 17648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:18.608144 17646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:18.609053 17640 server_base.cc:1034] running on GCE node
W20260924 01:03:18.609373 17645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:18.610054 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:18.611071 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:18.612300 17640 hybrid_clock.cc:648] HybridClock initialized: now 1790211798612263 us; error 41 us; skew 500 ppm
I20260924 01:03:18.612627 17640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:18 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:18.623162 17640 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:03:18.624893 17640 webserver.cc:460] Webserver started at http://127.8.254.65:41321/ using document root <none> and password file <none>
I20260924 01:03:18.625646 17640 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:18.625800 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:18.626130 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:18.629676 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "4c497d95ba494e3091559ec42ea63007"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "35544d27790038cfc11d63ad86dfdf65"
server_key_iv: "c87a689f1a04220d9831d150aad1eac7"
server_key_version: "encryptionkey@0"
I20260924 01:03:18.630693 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "4c497d95ba494e3091559ec42ea63007"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "35544d27790038cfc11d63ad86dfdf65"
server_key_iv: "c87a689f1a04220d9831d150aad1eac7"
server_key_version: "encryptionkey@0"
I20260924 01:03:18.636829 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 01:03:18.641212 17655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:18.642822 17640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:18.643018 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "4c497d95ba494e3091559ec42ea63007"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "35544d27790038cfc11d63ad86dfdf65"
server_key_iv: "c87a689f1a04220d9831d150aad1eac7"
server_key_version: "encryptionkey@0"
I20260924 01:03:18.643250 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:18.667011 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:18.671554 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:18.671872 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:18.672890 17640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:18.674404 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:18.674484 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:18.674667 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:18.674728 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:18.708760 17640 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:41909
I20260924 01:03:18.708806 17767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:41909 every 8 connection(s)
I20260924 01:03:18.710136 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:18.715566 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 17640
I20260924 01:03:18.715793 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:03:18.716579 9209 external_mini_cluster.cc:1428] Setting key 1f7e670d532a12e5eb374987acf5f54f
Sep 24 01:03:18 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211798, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:18.733762 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:18.712814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:41347 (local address 127.8.254.126:44031)
0924 01:03:18.713264 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:18.713268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:18.714214 (+ 946us) server_negotiation.cc:407] Connection header received
0924 01:03:18.716709 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:18.716715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:18.716809 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:18.716908 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:18.718770 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:18.719494 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:18.722154 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:18.722654 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:18.722762 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:18.727118 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:18.727126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:18.727135 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:18.727182 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:18.730670 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:18.731410 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:18.731418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:18.731425 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:18.731578 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:18.731991 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:18.731997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:18.732000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:18.732270 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:18.732430 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:18.733125 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 01:03:18.733371 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260924 01:03:18.734903 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:18.713272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:41347)
0924 01:03:18.714031 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:18.714077 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:18.716364 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:18.717205 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:18.717218 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:18.717865 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:18.718603 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:18.718622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:18.719654 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:18.719662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:18.721974 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:18.721992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:18.722901 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:18.722910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:18.723244 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:18.724033 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:18.724069 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:18.726876 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:18.730802 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:18.730810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:18.730825 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:18.731271 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:18.731683 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:18.731686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:18.731688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:18.731856 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:18.732403 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:18.732411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:18.732749 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 01:03:18.733020 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 01:03:18.733578 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":259,"threads_started":1}
I20260924 01:03:18.736590 17768 heartbeater.cc:347] Connected to a master server at 127.8.254.126:44031
I20260924 01:03:18.736905 17768 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:18.737632 17768 heartbeater.cc:511] Master 127.8.254.126:44031 requested a full tablet report, sending...
I20260924 01:03:18.740113 17579 ts_manager.cc:194] Registered new tserver with Master: 4c497d95ba494e3091559ec42ea63007 (127.8.254.65:41909)
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:03:18.741851 17579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:41347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:03:18.769282 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44031
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:19.107923 17774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:19.108232 17774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:19.108332 17774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:19.115844 17774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:19.115957 17774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:19.116070 17774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:19.126139 17774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:19.127465 17774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:19.128754 17774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:19.142495 17780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:19.142465 17782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:19.142529 17779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:19.143211 17774 server_base.cc:1034] running on GCE node
I20260924 01:03:19.143772 17774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:19.144670 17774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:19.145876 17774 hybrid_clock.cc:648] HybridClock initialized: now 1790211799145817 us; error 56 us; skew 500 ppm
I20260924 01:03:19.146174 17774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:19 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211799, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:19.149571 17774 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:03:19.151255 17774 webserver.cc:460] Webserver started at http://127.8.254.66:38499/ using document root <none> and password file <none>
I20260924 01:03:19.151990 17774 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:19.152124 17774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:19.152448 17774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:19.155568 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "ed54d48c150846e3b665998ea6c10b35"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "7c42c75fb10172105888cee247a44284"
server_key_iv: "d4f14cf4698f9b4dc7f3739a6aae88f9"
server_key_version: "encryptionkey@0"
I20260924 01:03:19.156386 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "ed54d48c150846e3b665998ea6c10b35"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "7c42c75fb10172105888cee247a44284"
server_key_iv: "d4f14cf4698f9b4dc7f3739a6aae88f9"
server_key_version: "encryptionkey@0"
I20260924 01:03:19.161690 17774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:19.165370 17789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:19.166842 17774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:19.167016 17774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "ed54d48c150846e3b665998ea6c10b35"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "7c42c75fb10172105888cee247a44284"
server_key_iv: "d4f14cf4698f9b4dc7f3739a6aae88f9"
server_key_version: "encryptionkey@0"
I20260924 01:03:19.167225 17774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:19.194738 17774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:19.198788 17774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:19.199098 17774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:19.200058 17774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:19.201606 17774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:19.201705 17774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:19.201835 17774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:19.201920 17774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:19.236238 17774 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:35087
I20260924 01:03:19.236276 17901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:35087 every 8 connection(s)
I20260924 01:03:19.237555 17774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:19.241837 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 17774
I20260924 01:03:19.242058 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:03:19.242534 9209 external_mini_cluster.cc:1428] Setting key 5668ed759b2b583a72a2e4c86d8e68ae
Sep 24 01:03:19 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211799, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:19.259948 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:19.240303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:43003 (local address 127.8.254.126:44031)
0924 01:03:19.240706 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:19.240711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:19.242043 (+ 1332us) server_negotiation.cc:407] Connection header received
0924 01:03:19.244030 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:19.244034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:19.244099 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:19.244191 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:19.245749 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.247033 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.248833 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.249286 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.249422 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:19.253427 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:19.253433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:19.253437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:19.253471 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:19.256180 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:19.257235 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:19.257239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:19.257242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:19.257369 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:19.257977 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:19.257980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:19.257982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:19.258322 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:19.258467 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:19.259464 (+ 997us) server_negotiation.cc:299] Negotiation successful
0924 01:03:19.259663 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":199,"threads_started":1}
I20260924 01:03:19.261654 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:19.240927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:43003)
0924 01:03:19.241767 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:19.241814 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:19.243719 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:19.244466 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:19.244476 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:19.245003 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:19.245569 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:19.245586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.247192 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.247197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:19.248663 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:19.248680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.249528 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.249536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:19.249777 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:19.250555 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:19.250580 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:19.253134 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:19.256335 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:19.256347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:19.256369 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:19.257105 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:19.257510 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:19.257517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:19.257523 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:19.257861 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:19.258456 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:19.258467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:19.258955 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 01:03:19.259291 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 01:03:19.259946 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":369,"threads_started":1}
I20260924 01:03:19.263391 17902 heartbeater.cc:347] Connected to a master server at 127.8.254.126:44031
I20260924 01:03:19.263722 17902 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:19.264410 17902 heartbeater.cc:511] Master 127.8.254.126:44031 requested a full tablet report, sending...
I20260924 01:03:19.265906 17579 ts_manager.cc:194] Registered new tserver with Master: ed54d48c150846e3b665998ea6c10b35 (127.8.254.66:35087)
I20260924 01:03:19.266795 17579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:43003
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:03:19.297406 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44031
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:19.630738 17908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:19.631017 17908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:19.631110 17908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:19.638772 17908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:19.638943 17908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:19.639210 17908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:19.649950 17908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:19.651281 17908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:19.652597 17908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:19.666321 17916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:19.666287 17914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:19.666741 17913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:19.667255 17908 server_base.cc:1034] running on GCE node
I20260924 01:03:19.667923 17908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:19.668922 17908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:19.670161 17908 hybrid_clock.cc:648] HybridClock initialized: now 1790211799670101 us; error 53 us; skew 500 ppm
I20260924 01:03:19.670535 17908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:19 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211799, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:19.673947 17908 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:03:19.675567 17908 webserver.cc:460] Webserver started at http://127.8.254.67:44407/ using document root <none> and password file <none>
I20260924 01:03:19.676262 17908 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:19.676407 17908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:19.676725 17908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:19.679680 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "9c5b817cc5334e3880ec50135798a185"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "c708f74f8666cb7c07a5f62e96c5fac1"
server_key_iv: "80fe8439bcf92ff59c33fe60b3082bcd"
server_key_version: "encryptionkey@0"
I20260924 01:03:19.680539 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "9c5b817cc5334e3880ec50135798a185"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "c708f74f8666cb7c07a5f62e96c5fac1"
server_key_iv: "80fe8439bcf92ff59c33fe60b3082bcd"
server_key_version: "encryptionkey@0"
I20260924 01:03:19.685796 17908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 01:03:19.689476 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:19.690950 17908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:19.691123 17908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "9c5b817cc5334e3880ec50135798a185"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "c708f74f8666cb7c07a5f62e96c5fac1"
server_key_iv: "80fe8439bcf92ff59c33fe60b3082bcd"
server_key_version: "encryptionkey@0"
I20260924 01:03:19.691334 17908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:19.723628 17908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:19.728165 17908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:19.728470 17908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:19.729463 17908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:19.730847 17908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:19.730955 17908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:19.731107 17908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:19.731189 17908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:19.745235 17768 heartbeater.cc:503] Master 127.8.254.126:44031 was elected leader, sending a full tablet report...
I20260924 01:03:19.765571 17908 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:42095
I20260924 01:03:19.765621 18035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:42095 every 8 connection(s)
I20260924 01:03:19.766990 17908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:19.770103 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 17908
I20260924 01:03:19.770360 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:03:19.771045 9209 external_mini_cluster.cc:1428] Setting key ed22dd65ac4ce1562d8fdc04bcefd0eb
Sep 24 01:03:19 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211799, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:19.787212 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:19.769543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:35405 (local address 127.8.254.126:44031)
0924 01:03:19.770023 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:19.770027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:19.771220 (+ 1193us) server_negotiation.cc:407] Connection header received
0924 01:03:19.773767 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:19.773771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:19.773853 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:19.773943 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:19.775560 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.776323 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.777822 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.778243 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.778344 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:19.781590 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:19.781594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:19.781597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:19.781631 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:19.784064 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:19.784762 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:19.784769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:19.784772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:19.784900 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:19.785334 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:19.785339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:19.785341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:19.785719 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:19.785886 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:19.786727 (+ 841us) server_negotiation.cc:299] Negotiation successful
0924 01:03:19.786933 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260924 01:03:19.788159 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:19.770145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:35405)
0924 01:03:19.771046 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:19.771092 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:19.773462 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:19.774272 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:19.774286 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:19.774852 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:19.775402 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:19.775418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.776486 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.776489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:19.777700 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:19.777711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.778457 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.778462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:19.778657 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:19.779240 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:19.779261 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:19.781390 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:19.784187 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:19.784193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:19.784205 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:19.784631 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:19.785039 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:19.785043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:19.785045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:19.785225 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:19.785878 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:19.785897 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:19.786313 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 01:03:19.786617 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 01:03:19.787043 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":358,"threads_started":1}
I20260924 01:03:19.789702 18036 heartbeater.cc:347] Connected to a master server at 127.8.254.126:44031
I20260924 01:03:19.790035 18036 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:19.790834 18036 heartbeater.cc:511] Master 127.8.254.126:44031 requested a full tablet report, sending...
I20260924 01:03:19.792382 17579 ts_manager.cc:194] Registered new tserver with Master: 9c5b817cc5334e3880ec50135798a185 (127.8.254.67:42095)
I20260924 01:03:19.793188 17579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:35405
I20260924 01:03:19.799055 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab.
Entry for principal oryx/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab.
WARNING: no policy specified for oryx/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab.
Entry for principal oryx/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab.
WARNING: no policy specified for oryx/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab.
Entry for principal oryx/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab.
I20260924 01:03:19.932747 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 17640
Sep 24 01:03:19 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:03:19.950109 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:19.934039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.8.254.65:41909)
0924 01:03:19.934753 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:19.934763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:19.934800 (+ 37us) server_negotiation.cc:407] Connection header received
0924 01:03:19.934861 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:19.934867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:19.935095 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:19.935307 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:19.936726 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.937751 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.939438 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:19.940038 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:19.940179 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:19.944300 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:19.944312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:19.944336 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:19.944373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:19.947227 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:19.947924 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:19.947932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:19.947934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:19.948062 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:19.948473 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:19.948476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:19.948478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:19.948818 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:19.949139 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:19.949476 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 01:03:19.949751 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":337,"threads_started":1}
I20260924 01:03:19.951438 17763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:20.035535 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 17640
I20260924 01:03:20.052296 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 17774
Sep 24 01:03:20 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:03:20.068249 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:20.053224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.8.254.66:35087)
0924 01:03:20.053761 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:20.053768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:20.053793 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:03:20.053855 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:20.053862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:20.054084 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:20.054265 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:20.055476 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:20.056445 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:20.057771 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:20.058363 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:20.058459 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:20.062639 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:20.062651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:20.062666 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:20.062706 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:20.065652 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:20.066317 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:20.066330 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:20.066337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:20.066460 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:20.066768 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:20.066771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:20.066773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:20.067118 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:20.067350 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:20.067655 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 01:03:20.067874 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":295,"threads_started":1}
I20260924 01:03:20.069512 17897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:20.155789 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 17774
I20260924 01:03:20.172044 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 17908
Sep 24 01:03:20 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:03:20.188961 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:20.173149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.8.254.67:42095)
0924 01:03:20.173847 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:20.173853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:20.173884 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:20.173958 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:20.173964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:20.174178 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:20.174442 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:20.175753 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:20.176759 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:20.178102 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:20.178970 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:20.179108 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:20.183402 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:20.183413 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:20.183430 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:20.183469 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:20.186484 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:20.187076 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:20.187083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:20.187085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:20.187222 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:20.187546 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:20.187549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:20.187551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:20.187867 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:20.188108 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:20.188336 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 01:03:20.188531 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":375,"threads_started":1}
I20260924 01:03:20.190294 18031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:20.272886 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 17908
I20260924 01:03:20.289222 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 17547
I20260924 01:03:20.290442 17609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:20.394310 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 17547
I20260924 01:03:20.415247 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44031
--webserver_interface=127.8.254.126
--webserver_port=45827
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:44031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:20.748533 18054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:20.748840 18054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:20.749012 18054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:20.756726 18054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:20.756847 18054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:20.756922 18054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:20.757025 18054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:20.757088 18054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:20.767629 18054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:44031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=45827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:20.769093 18054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:20.770558 18054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:20.782753 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:20.782697 18062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:20.782725 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:20.783905 18054 server_base.cc:1034] running on GCE node
I20260924 01:03:20.784741 18054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:20.786319 18054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:20.787595 18054 hybrid_clock.cc:648] HybridClock initialized: now 1790211800787544 us; error 37 us; skew 500 ppm
I20260924 01:03:20.787999 18054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:20 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211800, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:20.791282 18054 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:03:20.792913 18054 webserver.cc:460] Webserver started at http://127.8.254.126:45827/ using document root <none> and password file <none>
I20260924 01:03:20.793659 18054 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:20.793824 18054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:20.799961 18054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 01:03:20.803032 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:20.804450 18054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:20.804749 18054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "c908fb4b3dba4f228070043b7c62169e"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "d75e38cd02fde52226675ac08e6ded52"
server_key_iv: "07a329f1d4dc7c13896ef9b63fc16fb5"
server_key_version: "encryptionkey@0"
I20260924 01:03:20.805888 18054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:20.845579 18054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:20.850097 18054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:20.850498 18054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:20.871331 18054 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:44031
I20260924 01:03:20.871352 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:44031 every 8 connection(s)
I20260924 01:03:20.873032 18054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:03:20.877815 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18054
I20260924 01:03:20.878801 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:41909
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:44031
--webserver_port=41321
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:03:20.881192 18121 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:03:20.886479 18121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Bootstrap starting.
I20260924 01:03:20.905699 18121 log.cc:824] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:20.915198 18121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:03:20.915892 18121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Bootstrap complete.
I20260924 01:03:20.925016 18121 raft_consensus.cc:348] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:20.925931 18121 raft_consensus.cc:729] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c908fb4b3dba4f228070043b7c62169e, State: Initialized, Role: FOLLOWER
I20260924 01:03:20.926739 18121 consensus_queue.cc:260] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:20.926987 18121 raft_consensus.cc:388] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:03:20.927110 18121 raft_consensus.cc:482] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:03:20.927296 18121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 1 FOLLOWER]: Advancing to term 2
I20260924 01:03:20.931238 18121 raft_consensus.cc:504] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:20.931794 18121 leader_election.cc:302] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c908fb4b3dba4f228070043b7c62169e; no voters:
I20260924 01:03:20.932410 18121 leader_election.cc:288] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 01:03:20.932633 18125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 01:03:20.933730 18125 raft_consensus.cc:686] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [term 2 LEADER]: Becoming Leader. State: Replica: c908fb4b3dba4f228070043b7c62169e, State: Running, Role: LEADER
I20260924 01:03:20.934371 18125 consensus_queue.cc:237] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } }
I20260924 01:03:20.934810 18121 sys_catalog.cc:564] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:20.936896 18127 sys_catalog.cc:455] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c908fb4b3dba4f228070043b7c62169e. Latest consensus state: current_term: 2 leader_uuid: "c908fb4b3dba4f228070043b7c62169e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } } }
I20260924 01:03:20.937254 18127 sys_catalog.cc:458] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:20.937788 18126 sys_catalog.cc:455] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c908fb4b3dba4f228070043b7c62169e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908fb4b3dba4f228070043b7c62169e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44031 } } }
I20260924 01:03:20.938125 18126 sys_catalog.cc:458] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:20.945779 18130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:20.955323 18130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:20.957185 18130 catalog_manager.cc:1259] Loaded cluster ID: ee7aae3f84c54cdd9e234d2c6e1927e7
I20260924 01:03:20.957324 18130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:20.962659 18130 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:20.964668 18130 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c908fb4b3dba4f228070043b7c62169e: Loaded TSK: 0
I20260924 01:03:20.967070 18130 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 01:03:21.248359 18123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:21.248669 18123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:21.248867 18123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:21.256481 18123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:21.256599 18123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:21.256771 18123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:21.267190 18123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.65.keytab
--principal=oryx/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:41909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=41321
--tserver_master_addrs=127.8.254.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:21.268622 18123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:21.269958 18123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:21.284206 18148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:21.284227 18149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:21.284314 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:21.285921 18123 server_base.cc:1034] running on GCE node
I20260924 01:03:21.286695 18123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:21.287791 18123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:21.289040 18123 hybrid_clock.cc:648] HybridClock initialized: now 1790211801288968 us; error 64 us; skew 500 ppm
I20260924 01:03:21.289399 18123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:21 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211801, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:21.293009 18123 init.cc:377] Logged in from keytab as oryx/127.8.254.65@KRBTEST.COM (short username oryx)
I20260924 01:03:21.294852 18123 webserver.cc:460] Webserver started at http://127.8.254.65:41321/ using document root <none> and password file <none>
I20260924 01:03:21.295770 18123 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:21.295919 18123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:21.302965 18123 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 01:03:21.306061 18158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:21.307749 18123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:03:21.307914 18123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "4c497d95ba494e3091559ec42ea63007"
format_stamp: "Formatted at 2026-09-24 01:03:18 on dist-test-slave-5c7l"
server_key: "35544d27790038cfc11d63ad86dfdf65"
server_key_iv: "c87a689f1a04220d9831d150aad1eac7"
server_key_version: "encryptionkey@0"
I20260924 01:03:21.309250 18123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:21.328812 18123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:21.334254 18123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:21.334560 18123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:21.335752 18123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:21.337458 18123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:21.337566 18123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:21.337677 18123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:21.337749 18123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:21.377521 18123 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:41909
I20260924 01:03:21.377539 18270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:41909 every 8 connection(s)
I20260924 01:03:21.379107 18123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:21.384632 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18123
I20260924 01:03:21.385375 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:35087
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:44031
--webserver_port=38499
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:03:21 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:21 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 01:03:21.725771 18275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:21.726120 18275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:21.726310 18275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:21.733656 18275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:21.733811 18275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:21.733984 18275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:21.744268 18275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.66.keytab
--principal=oryx/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:35087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=38499
--tserver_master_addrs=127.8.254.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:21.745666 18275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:21.747128 18275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:21.765911 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:21.765955 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:21.766762 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:21.772143 18275 server_base.cc:1034] running on GCE node
I20260924 01:03:21.772864 18275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:21.773958 18275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:21.775182 18275 hybrid_clock.cc:648] HybridClock initialized: now 1790211801775133 us; error 49 us; skew 500 ppm
I20260924 01:03:21.775512 18275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:21 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211801, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:21.779174 18275 init.cc:377] Logged in from keytab as oryx/127.8.254.66@KRBTEST.COM (short username oryx)
I20260924 01:03:21.780921 18275 webserver.cc:460] Webserver started at http://127.8.254.66:38499/ using document root <none> and password file <none>
I20260924 01:03:21.781718 18275 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:21.781876 18275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:21.787480 18275 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 01:03:21.790663 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:21.792591 18275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 01:03:21.792791 18275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "ed54d48c150846e3b665998ea6c10b35"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "7c42c75fb10172105888cee247a44284"
server_key_iv: "d4f14cf4698f9b4dc7f3739a6aae88f9"
server_key_version: "encryptionkey@0"
I20260924 01:03:21.793809 18275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:21.825981 18275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:21.830592 18275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:21.830883 18275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:21.832051 18275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:21.833511 18275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:21.833630 18275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:21.833751 18275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:21.833832 18275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:21.882645 18275 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:35087
I20260924 01:03:21.882863 18403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:35087 every 8 connection(s)
I20260924 01:03:21.884542 18275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:21.893853 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18275
I20260924 01:03:21.894893 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:42095
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:44031
--webserver_port=44407
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:03:21 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:21 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 01:03:22.259021 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:22.259352 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:22.259524 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:22.267216 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:22.267369 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:22.267565 18408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:22.278874 18408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/krb5kdc/oryx_127.8.254.67.keytab
--principal=oryx/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:42095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=44407
--tserver_master_addrs=127.8.254.126:44031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:22.280434 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:22.281803 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:22.301720 18415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:22.302743 18414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:22.311959 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:22.312719 18408 server_base.cc:1034] running on GCE node
I20260924 01:03:22.313874 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:22.315709 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:22.317214 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790211802317113 us; error 81 us; skew 500 ppm
I20260924 01:03:22.317792 18408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211802, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:22.324155 18408 init.cc:377] Logged in from keytab as oryx/127.8.254.67@KRBTEST.COM (short username oryx)
I20260924 01:03:22.328708 18408 webserver.cc:460] Webserver started at http://127.8.254.67:44407/ using document root <none> and password file <none>
I20260924 01:03:22.330107 18408 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:22.330359 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:22.341678 18408 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 01:03:22.347453 18424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:22.350564 18408 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260924 01:03:22.350850 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "9c5b817cc5334e3880ec50135798a185"
format_stamp: "Formatted at 2026-09-24 01:03:19 on dist-test-slave-5c7l"
server_key: "c708f74f8666cb7c07a5f62e96c5fac1"
server_key_iv: "80fe8439bcf92ff59c33fe60b3082bcd"
server_key_version: "encryptionkey@0"
I20260924 01:03:22.353063 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:22.392753 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:22.401118 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:22.401558 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:22.403330 18408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:22.405742 18408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:22.405896 18408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:22.406066 18408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:22.406176 18408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:22.466594 18408 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:42095
I20260924 01:03:22.466830 18536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:42095 every 8 connection(s)
I20260924 01:03:22.469374 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:22.475982 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18408
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:22.521342 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:22.478384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.8.254.126:44031)
0924 01:03:22.487225 (+ 8841us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:22.487234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:22.487260 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:03:22.487371 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:22.487380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:22.487502 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:22.490792 (+ 3290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:22.491195 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.492976 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.497630 (+ 4654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.498789 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.498977 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.504563 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:22.504580 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:22.504611 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:22.504692 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:22.511307 (+ 6615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:22.512690 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:22.512705 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:22.512719 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:22.513040 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:22.513771 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:22.513779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:22.513786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:22.516122 (+ 2336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:22.516569 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:22.519000 (+ 2431us) server_negotiation.cc:299] Negotiation successful
0924 01:03:22.519748 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8701,"thread_start_us":8253,"threads_started":1}
W20260924 01:03:22.838636 18274 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.445s user 0.005s sys 0.014s
W20260924 01:03:22.840929 18273 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:21.382082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:47927 (local address 127.8.254.126:44031)
0924 01:03:21.382920 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:21.382936 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:21.383219 (+ 283us) server_negotiation.cc:407] Connection header received
0924 01:03:21.385317 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:21.385336 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:21.385791 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:21.386212 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:21.387585 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.389556 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.391133 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.392240 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.392369 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.840407 (+1448038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:47927: BlockingRecv error: recv got EOF from 127.8.254.65:47927 (error 108)
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":266,"threads_started":1}
I20260924 01:03:22.841643 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:21.382275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:47927)
0924 01:03:21.383059 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:21.383095 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:21.384680 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:21.386341 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:21.386349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:21.386849 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:21.387376 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:21.387393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.389706 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.389709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:21.390993 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:21.391005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.392464 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.392473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:21.392769 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:21.393523 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:21.393545 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:22.839982 (+1446437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":345,"threads_started":1}
W20260924 01:03:22.843632 18271 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:44031 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.254.126:44031: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:22.861480 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:22.844632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:60977)
0924 01:03:22.845035 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:22.845062 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:22.845239 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:22.845725 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:22.845731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:22.846277 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:22.846578 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:22.846596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.848781 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.848791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:22.851172 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:22.851196 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.852621 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.852630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:22.852905 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.853968 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:22.854006 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:22.860998 (+ 6992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206}
W20260924 01:03:22.861946 18273 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:22.844828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:60977 (local address 127.8.254.126:44031)
0924 01:03:22.845062 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:22.845072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:22.845141 (+ 69us) server_negotiation.cc:407] Connection header received
0924 01:03:22.845410 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:22.845418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:22.845536 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:22.845699 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:22.846781 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.848566 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.851390 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.852361 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.852545 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.861420 (+ 8875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:60977: BlockingRecv error: recv got EOF from 127.8.254.65:60977 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:22 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 01:03:22.886588 18273 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:22.863425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:35651 (local address 127.8.254.126:44031)
0924 01:03:22.863671 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:22.863681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:22.864006 (+ 325us) server_negotiation.cc:407] Connection header received
0924 01:03:22.864388 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:22.864397 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:22.864544 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:22.864693 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:22.866433 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.868117 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.874832 (+ 6715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.875924 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.876069 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.886282 (+ 10213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:35651: BlockingRecv error: recv got EOF from 127.8.254.65:35651 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 01:03:22.887228 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:22.862978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:35651)
0924 01:03:22.863860 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:22.863888 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:22.864167 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:22.864740 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:22.864748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:22.865634 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:22.866087 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:22.866126 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.868381 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.868391 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:22.874542 (+ 6151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:22.874574 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.876498 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.876508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:22.876810 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.877850 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:22.877887 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:22.886901 (+ 9014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":661,"threads_started":1}
W20260924 01:03:22.887847 18271 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:03:23.204324 18407 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.306s user 0.001s sys 0.012s
I20260924 01:03:23.205962 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:21.888983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:41565)
0924 01:03:21.889830 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:21.889884 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:21.891644 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:21.892443 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:21.892451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:21.893010 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:21.893587 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:21.893604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.894857 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.894863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:21.896294 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:21.896319 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.897157 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.897166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:21.897471 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:21.898184 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:21.898236 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.205118 (+1306882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
W20260924 01:03:23.206039 18406 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:21.888182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:41565 (local address 127.8.254.126:44031)
0924 01:03:21.889127 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:21.889137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:21.890012 (+ 875us) server_negotiation.cc:407] Connection header received
0924 01:03:21.891946 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:21.891955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:21.892078 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:21.892261 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:21.893781 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.894690 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.896485 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:21.896888 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:21.897019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.205543 (+1308524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:41565: BlockingRecv error: recv got EOF from 127.8.254.66:41565 (error 108)
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":566,"threads_started":1}
W20260924 01:03:23.207494 18404 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:44031 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.254.126:44031: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:23.217460 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:23.208371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:42345)
0924 01:03:23.208584 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:23.208600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:23.208744 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:23.209121 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:23.209126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:23.209580 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:23.209771 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.209780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.210942 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.210948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.211898 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.211911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.212629 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.212635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.212807 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.213516 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:23.213534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.217151 (+ 3617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 01:03:23.217610 18406 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:23.208385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:42345 (local address 127.8.254.126:44031)
0924 01:03:23.208595 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:23.208605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:23.208685 (+ 80us) server_negotiation.cc:407] Connection header received
0924 01:03:23.208880 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:23.208885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:23.208971 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:23.209062 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:23.209911 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.210808 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.212046 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.212460 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.212588 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.217369 (+ 4781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:42345: BlockingRecv error: recv got EOF from 127.8.254.66:42345 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:23.228011 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:23.218503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:49937)
0924 01:03:23.218671 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:23.218690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:23.218790 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:23.219458 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:23.219465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:23.219831 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:23.220060 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.220076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.221200 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.221205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.222278 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.222292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.222958 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.222964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.223179 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.223752 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:23.223770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.227693 (+ 3923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 01:03:23.228171 18406 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:23.218821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:49937 (local address 127.8.254.126:44031)
0924 01:03:23.219056 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:23.219066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:23.219091 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:03:23.219168 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:23.219177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:23.219283 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:23.219431 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:23.220202 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.221051 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.222410 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.222806 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.222934 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.227911 (+ 4977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:49937: BlockingRecv error: recv got EOF from 127.8.254.66:49937 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 01:03:23.228364 18404 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:03:23.728773 18539 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.214s user 0.000s sys 0.025s
W20260924 01:03:23.729977 18542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:22.477013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:56693 (local address 127.8.254.126:44031)
0924 01:03:22.486829 (+ 9816us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:22.486840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:22.496922 (+ 10082us) server_negotiation.cc:407] Connection header received
0924 01:03:22.501207 (+ 4285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:22.501217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:22.501363 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:22.501472 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:22.505032 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.507180 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.510934 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.511916 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.512050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.729744 (+1217694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:56693: BlockingRecv error: recv got EOF from 127.8.254.67:56693 (error 108)
Metrics: {"server-negotiator.queue_time_us":9629,"thread_start_us":9947,"threads_started":1}
I20260924 01:03:23.730561 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:22.474646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:56693)
0924 01:03:22.496622 (+ 21976us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:22.496701 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:22.500732 (+ 4031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:22.502390 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:22.502410 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:22.503552 (+ 1142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:22.504742 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:22.504775 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.507522 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.507531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:22.510680 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:22.510704 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:22.512581 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:22.512591 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:22.513037 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:22.514494 (+ 1457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:22.514539 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.729569 (+1215030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21456,"thread_start_us":21098,"threads_started":1}
W20260924 01:03:23.731508 18537 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:44031 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.254.126:44031: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:23.746775 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:23.732214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:56639)
0924 01:03:23.732419 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:23.732440 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:23.732578 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:23.732967 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:23.732972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:23.733433 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:23.733608 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.733616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.734657 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.734661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.735658 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.735670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.736398 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.736403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.736533 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.742446 (+ 5913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:23.742467 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.746559 (+ 4092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 01:03:23.747064 18542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:23.732292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:56639 (local address 127.8.254.126:44031)
0924 01:03:23.732450 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:23.732454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:23.732513 (+ 59us) server_negotiation.cc:407] Connection header received
0924 01:03:23.732722 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:23.732731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:23.732803 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:23.732881 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:23.733737 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.734529 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.735775 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.736248 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.736342 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.746767 (+ 10425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:56639: BlockingRecv error: recv got EOF from 127.8.254.67:56639 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:23.755270 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:23.747581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:48285)
0924 01:03:23.747733 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:23.747748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:23.747833 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:23.748164 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:23.748167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:23.748508 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:23.748663 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.748671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.749563 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.749566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.750532 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.750543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.751183 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.751187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.751304 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.751805 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:23.751822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.755065 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:03:23.755581 18537 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:03:23.755643 18542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:23.747720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:48285 (local address 127.8.254.126:44031)
0924 01:03:23.747925 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:23.747929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:23.747944 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:23.747985 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:23.747988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:23.748045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:23.748122 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:23.748776 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.749444 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.750647 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.751063 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.751162 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.755270 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:48285: BlockingRecv error: recv got EOF from 127.8.254.67:48285 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:23 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:23.897658 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:23.888388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:47173)
0924 01:03:23.888951 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:23.888966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:23.889080 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:23.889398 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:23.889401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:23.889894 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:23.890119 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.890130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.891337 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.891343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.892334 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:23.892346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.892986 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.892990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:23.893109 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.893712 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:23.893730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:23.897388 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":344,"threads_started":1}
W20260924 01:03:23.897851 18542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:23.888475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:47173 (local address 127.8.254.126:44031)
0924 01:03:23.888636 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:23.888640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:23.889035 (+ 395us) server_negotiation.cc:407] Connection header received
0924 01:03:23.889191 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:23.889194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:23.889273 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:23.889351 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:23.890284 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.891191 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.892483 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:23.892861 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:23.892956 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:23.897652 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:47173: BlockingRecv error: recv got EOF from 127.8.254.65:47173 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 01:03:24 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:24 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:24.238559 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:24.229051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:54779)
0924 01:03:24.229659 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:24.229675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:24.229833 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:24.230224 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:24.230229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:24.230680 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:24.230827 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:24.230835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.231968 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.231974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:24.232911 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:24.232922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.233711 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.233717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:24.233883 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:24.234543 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:24.234560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:24.238189 (+ 3629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":358,"threads_started":1}
W20260924 01:03:24.238735 18542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:24.229146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54779 (local address 127.8.254.126:44031)
0924 01:03:24.229381 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:24.229388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:24.229767 (+ 379us) server_negotiation.cc:407] Connection header received
0924 01:03:24.229960 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:24.229966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:24.230052 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:24.230155 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:24.230979 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.231828 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.233061 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.233526 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.233653 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:24.238467 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:54779: BlockingRecv error: recv got EOF from 127.8.254.66:54779 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 01:03:24 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:24 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:24.765053 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:24.756136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:49729)
0924 01:03:24.756687 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:24.756704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:24.756812 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:24.757282 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:24.757286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:24.757715 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:24.757860 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:24.757868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.758891 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.758895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:24.759857 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:24.759869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.760579 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.760584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:24.760703 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:24.761277 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:24.761297 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:24.764791 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":324,"threads_started":1}
W20260924 01:03:24.765282 18548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:24.756321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:49729 (local address 127.8.254.126:44031)
0924 01:03:24.757008 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:24.757012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:24.757028 (+ 16us) server_negotiation.cc:407] Connection header received
0924 01:03:24.757077 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:24.757080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:24.757144 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:24.757234 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:24.757992 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.758759 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.759979 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.760391 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.760483 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:24.765009 (+ 4526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:49729: BlockingRecv error: recv got EOF from 127.8.254.67:49729 (error 108)
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":489,"threads_started":1}
Sep 24 01:03:24 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:24 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:24.907398 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:24.898704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:40549)
0924 01:03:24.899181 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:24.899196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:24.899310 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:24.899607 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:24.899610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:24.899943 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:24.900160 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:24.900176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.901534 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.901554 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:24.902564 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:24.902577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.903211 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.903215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:24.903337 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:24.903914 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:24.903930 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:24.907159 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
W20260924 01:03:24.907541 18548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:24.898791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:40549 (local address 127.8.254.126:44031)
0924 01:03:24.899026 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:24.899033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:24.899274 (+ 241us) server_negotiation.cc:407] Connection header received
0924 01:03:24.899408 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:24.899412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:24.899484 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:24.899567 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:24.900324 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.901377 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.902690 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:24.903093 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:24.903187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:24.907346 (+ 4159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:40549: BlockingRecv error: recv got EOF from 127.8.254.65:40549 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 01:03:25 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:25 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:25.248978 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:25.239793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:54679)
0924 01:03:25.240328 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:25.240344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:25.240477 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:25.240803 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:25.240806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:25.241215 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:25.241394 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:25.241402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.242742 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.242749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:25.243873 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:25.243886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.244549 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.244553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:25.244672 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:25.245229 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:25.245246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:25.248680 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":315,"threads_started":1}
W20260924 01:03:25.249126 18548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:25.239922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54679 (local address 127.8.254.126:44031)
0924 01:03:25.240112 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:25.240117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:25.240421 (+ 304us) server_negotiation.cc:407] Connection header received
0924 01:03:25.240585 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:25.240589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:25.240667 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:25.240760 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:25.241547 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.242591 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.243997 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.244418 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.244512 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:25.248860 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:54679: BlockingRecv error: recv got EOF from 127.8.254.66:54679 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 01:03:25 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:25 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:25.775574 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:25.766040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:37505)
0924 01:03:25.766653 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:25.766671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:25.766789 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:25.767262 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:25.767265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:25.767652 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:25.767828 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:25.767836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.769207 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.769211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:25.770350 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:25.770363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.771114 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.771120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:25.771302 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:25.771838 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:25.771856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:25.775325 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":408,"threads_started":1}
W20260924 01:03:25.775982 18552 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:25.766398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:37505 (local address 127.8.254.126:44031)
0924 01:03:25.766970 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:25.766975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:25.766990 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:25.767043 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:25.767046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:25.767119 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:25.767211 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:25.767997 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.769046 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.770500 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.770953 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.771097 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:25.775541 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:37505: BlockingRecv error: recv got EOF from 127.8.254.67:37505 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":355,"threads_started":1}
Sep 24 01:03:25 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:25 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:25.917974 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:25.908263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:46585)
0924 01:03:25.908752 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:25.908767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:25.908920 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:25.909300 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:25.909305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:25.909733 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:25.909905 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:25.909913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.911089 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.911095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:25.911991 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:25.912002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.912736 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.912741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:25.912931 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:25.913537 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:25.913556 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:25.917638 (+ 4082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":247,"threads_started":1}
W20260924 01:03:25.918061 18552 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:25.908334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:46585 (local address 127.8.254.126:44031)
0924 01:03:25.908576 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:25.908582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:25.908851 (+ 269us) server_negotiation.cc:407] Connection header received
0924 01:03:25.909043 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:25.909048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:25.909136 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:25.909273 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:25.910064 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.910930 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.912154 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:25.912565 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:25.912710 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:25.917819 (+ 5109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:46585: BlockingRecv error: recv got EOF from 127.8.254.65:46585 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 01:03:26 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:26 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:26.259280 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:26.249993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:44667)
0924 01:03:26.250563 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:26.250579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:26.250751 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:26.251226 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:26.251231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:26.251753 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:26.251929 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:26.251939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.252941 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.252945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:26.253912 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:26.253925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.254667 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.254672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:26.254811 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:26.255505 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:26.255523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:26.259019 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":274,"threads_started":1}
W20260924 01:03:26.259501 18552 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:26.250075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:44667 (local address 127.8.254.126:44031)
0924 01:03:26.250303 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:26.250311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:26.250674 (+ 363us) server_negotiation.cc:407] Connection header received
0924 01:03:26.250893 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:26.250925 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:26.251015 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:26.251141 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:26.252078 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.252807 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.254054 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.254508 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.254605 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:26.259226 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:44667: BlockingRecv error: recv got EOF from 127.8.254.66:44667 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 01:03:26 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:26 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:26.785492 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:26.776513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:43547)
0924 01:03:26.777054 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:26.777072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:26.777176 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:26.777508 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:26.777511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:26.777852 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:26.778020 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:26.778028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.779179 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.779184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:26.780062 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:26.780074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.780764 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.780769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:26.780889 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:26.781530 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:26.781548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:26.785129 (+ 3581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
W20260924 01:03:26.785599 18556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:26.776661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:43547 (local address 127.8.254.126:44031)
0924 01:03:26.777144 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:26.777148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:26.777165 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:03:26.777290 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:26.777294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:26.777368 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:26.777461 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:26.778158 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.779039 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.780197 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.780629 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.780727 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:26.785342 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:43547: BlockingRecv error: recv got EOF from 127.8.254.67:43547 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":297,"threads_started":1}
Sep 24 01:03:26 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:26 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:26.928153 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:26.918961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:39915)
0924 01:03:26.919483 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:26.919499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:26.919608 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:26.919902 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:26.919905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:26.920308 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:26.920471 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:26.920479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.921609 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.921613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:26.922740 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:26.922755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.923523 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.923528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:26.923659 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:26.924299 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:26.924317 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:26.927869 (+ 3552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":309,"threads_started":1}
W20260924 01:03:26.928436 18556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:26.919080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:39915 (local address 127.8.254.126:44031)
0924 01:03:26.919301 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:26.919306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:26.919577 (+ 271us) server_negotiation.cc:407] Connection header received
0924 01:03:26.919707 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:26.919711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:26.919772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:26.919852 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:26.920706 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.921472 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.922939 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:26.923375 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:26.923488 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:26.928147 (+ 4659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:39915: BlockingRecv error: recv got EOF from 127.8.254.65:39915 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 01:03:27 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:27 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:27.270495 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:27.260683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:53053)
0924 01:03:27.261484 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:27.261528 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:27.261693 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:27.262187 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:27.262213 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:27.262724 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:27.262972 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:27.262986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.264167 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.264171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:27.265154 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:27.265166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.265860 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.265864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:27.265993 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:27.266659 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:27.266678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:27.270170 (+ 3492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":484,"threads_started":1}
W20260924 01:03:27.270742 18556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:27.260755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:53053 (local address 127.8.254.126:44031)
0924 01:03:27.261017 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:27.261026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:27.261641 (+ 615us) server_negotiation.cc:407] Connection header received
0924 01:03:27.261865 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:27.261870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:27.261955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:27.262069 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:27.263136 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.264041 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.265299 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.265726 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.265820 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:27.270419 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:53053: BlockingRecv error: recv got EOF from 127.8.254.66:53053 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 01:03:27 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:27 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:27.796393 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:27.786726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:42325)
0924 01:03:27.787392 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:27.787412 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:27.787527 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:27.787898 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:27.787903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:27.788292 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:27.788497 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:27.788508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.789632 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.789637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:27.790843 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:27.790858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.791793 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.791804 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:27.792009 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:27.792708 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:27.792726 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:27.796144 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":282,"threads_started":1}
W20260924 01:03:27.796663 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:27.786949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:42325 (local address 127.8.254.126:44031)
0924 01:03:27.787442 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:27.787447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:27.787465 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:03:27.787651 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:27.787655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:27.787731 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:27.787822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:27.788711 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.789490 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.791022 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.791621 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.791774 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:27.796339 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:42325: BlockingRecv error: recv got EOF from 127.8.254.67:42325 (error 108)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":266,"threads_started":1}
Sep 24 01:03:27 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:27 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:27.937857 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:27.929108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:46411)
0924 01:03:27.929663 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:27.929678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:27.929787 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:27.930076 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:27.930080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:27.930502 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:27.930650 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:27.930658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.931654 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.931657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:27.932595 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:27.932606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.933242 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.933246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:27.933385 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:27.933999 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:27.934016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:27.937639 (+ 3623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":330,"threads_started":1}
W20260924 01:03:27.938109 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:27.929156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:46411 (local address 127.8.254.126:44031)
0924 01:03:27.929328 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:27.929333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:27.929748 (+ 415us) server_negotiation.cc:407] Connection header received
0924 01:03:27.929882 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:27.929885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:27.929947 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:27.930028 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:27.930770 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.931528 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.932723 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:27.933103 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:27.933193 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:27.937833 (+ 4640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:46411: BlockingRecv error: recv got EOF from 127.8.254.65:46411 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 01:03:28 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:28 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:28.280936 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:28.271550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:58879)
0924 01:03:28.272160 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:28.272178 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:28.272300 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:28.272804 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:28.272808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:28.273236 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:28.273467 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:28.273478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.274503 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.274507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:28.275582 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:28.275595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.276324 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.276329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:28.276497 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:28.277101 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:28.277120 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:28.280648 (+ 3528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":337,"threads_started":1}
W20260924 01:03:28.281261 18560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:28.271707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:58879 (local address 127.8.254.126:44031)
0924 01:03:28.271884 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:28.271889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:28.272256 (+ 367us) server_negotiation.cc:407] Connection header received
0924 01:03:28.272483 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:28.272486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:28.272547 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:28.272635 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:28.273605 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.274357 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.275715 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.276156 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.276247 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:28.280946 (+ 4699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:58879: BlockingRecv error: recv got EOF from 127.8.254.66:58879 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 01:03:28 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:28 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:28.816705 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:28.797599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:35185)
0924 01:03:28.798179 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:28.798241 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:28.798435 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:28.798809 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:28.798812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:28.799236 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:28.799384 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:28.799392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.800371 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.800375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:28.801450 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:28.801462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.802276 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.802282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:28.802404 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:28.812572 (+ 10168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:28.812594 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:28.816381 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":279,"threads_started":1}
W20260924 01:03:28.816910 18564 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:28.797659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:35185 (local address 127.8.254.126:44031)
0924 01:03:28.798242 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:28.798249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:28.798370 (+ 121us) server_negotiation.cc:407] Connection header received
0924 01:03:28.798584 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:28.798588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:28.798669 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:28.798759 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:28.799532 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.800255 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.801624 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.802074 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.802210 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:28.816644 (+ 14434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:35185: BlockingRecv error: recv got EOF from 127.8.254.67:35185 (error 108)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
Sep 24 01:03:28 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:28 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:28.948590 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:28.939208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:55429)
0924 01:03:28.939753 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:28.939769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:28.939931 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:28.940325 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:28.940330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:28.940784 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:28.940929 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:28.940937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.942014 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.942019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:28.942973 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:28.942984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.943714 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.943719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:28.943900 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:28.944555 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:28.944573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:28.948257 (+ 3684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":295,"threads_started":1}
W20260924 01:03:28.948947 18564 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:28.939268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:55429 (local address 127.8.254.126:44031)
0924 01:03:28.939479 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:28.939485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:28.939867 (+ 382us) server_negotiation.cc:407] Connection header received
0924 01:03:28.940095 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:28.940100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:28.940175 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:28.940287 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:28.941096 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.941841 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.943123 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:28.943546 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:28.943692 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:28.948482 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:55429: BlockingRecv error: recv got EOF from 127.8.254.65:55429 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 01:03:29 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:29 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:29.291826 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:29.282434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:59207)
0924 01:03:29.282924 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:29.282940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:29.283064 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:29.283513 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:29.283517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:29.283915 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:29.284159 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:29.284174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.285368 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.285376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:29.286413 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:29.286427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.287173 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.287180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:29.287367 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:29.288035 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:29.288053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:29.291494 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":290,"threads_started":1}
W20260924 01:03:29.291951 18564 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:29.282601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:59207 (local address 127.8.254.126:44031)
0924 01:03:29.282883 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:29.282890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:29.283032 (+ 142us) server_negotiation.cc:407] Connection header received
0924 01:03:29.283221 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:29.283226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:29.283323 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:29.283506 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:29.284320 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.285218 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.286578 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.286993 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.287134 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:29.291673 (+ 4539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:59207: BlockingRecv error: recv got EOF from 127.8.254.66:59207 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 24 01:03:29 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:29 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:29.827611 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:29.817823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:39025)
0924 01:03:29.818423 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:29.818446 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:29.818587 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:29.818982 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:29.818987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:29.819524 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:29.819776 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:29.819789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.821235 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.821240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:29.822242 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:29.822255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.823039 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.823043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:29.823178 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:29.823791 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:29.823812 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:29.827378 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
W20260924 01:03:29.827960 18568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:29.817911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:39025 (local address 127.8.254.126:44031)
0924 01:03:29.818570 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:29.818576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:29.818596 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:03:29.818750 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:29.818754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:29.818825 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:29.818945 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:29.819955 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.821053 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.822442 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.822896 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.822999 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:29.827632 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:39025: BlockingRecv error: recv got EOF from 127.8.254.67:39025 (error 108)
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":436,"threads_started":1}
Sep 24 01:03:29 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:29 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:29.959228 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:29.949597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:37123)
0924 01:03:29.950224 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:29.950240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:29.950360 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:29.950689 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:29.950692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:29.951168 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:29.951352 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:29.951360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.952376 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.952380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:29.953344 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:29.953357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.954136 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.954144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:29.954370 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:29.955098 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:29.955121 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:29.958963 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":271,"threads_started":1}
W20260924 01:03:29.959404 18568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:29.949663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:37123 (local address 127.8.254.126:44031)
0924 01:03:29.949892 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:29.949901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:29.950327 (+ 426us) server_negotiation.cc:407] Connection header received
0924 01:03:29.950466 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:29.950470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:29.950536 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:29.950638 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:29.951485 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.952224 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.953482 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:29.953939 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:29.954068 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:29.959168 (+ 5100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:37123: BlockingRecv error: recv got EOF from 127.8.254.65:37123 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 01:03:30 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:30 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:30.302063 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:30.292939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:35047)
0924 01:03:30.293532 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:30.293548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:30.293662 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:30.293981 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:30.293984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:30.294430 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:30.294611 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:30.294620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.295570 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.295574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:30.296486 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:30.296497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.297266 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.297271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:30.297382 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:30.297947 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:30.297966 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:30.301726 (+ 3760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":270,"threads_started":1}
W20260924 01:03:30.302245 18568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:30.292976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:35047 (local address 127.8.254.126:44031)
0924 01:03:30.293255 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:30.293265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:30.293635 (+ 370us) server_negotiation.cc:407] Connection header received
0924 01:03:30.293765 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:30.293769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:30.293835 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:30.293916 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:30.294736 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.295451 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.296613 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.297117 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.297212 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:30.301976 (+ 4764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:35047: BlockingRecv error: recv got EOF from 127.8.254.66:35047 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 24 01:03:30 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:30 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:30.838311 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:30.828719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:52325)
0924 01:03:30.829367 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:30.829384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:30.829511 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:30.829867 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:30.829870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:30.830410 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:30.830649 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:30.830662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.831986 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.831992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:30.833128 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:30.833141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.833844 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.833848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:30.833961 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:30.834592 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:30.834609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:30.837994 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":335,"threads_started":1}
W20260924 01:03:30.838547 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:30.828853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:52325 (local address 127.8.254.126:44031)
0924 01:03:30.829367 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:30.829371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:30.829468 (+ 97us) server_negotiation.cc:407] Connection header received
0924 01:03:30.829641 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:30.829645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:30.829714 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:30.829815 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:30.830831 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.831835 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.833262 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.833707 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.833804 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:30.838273 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:52325: BlockingRecv error: recv got EOF from 127.8.254.67:52325 (error 108)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":320,"threads_started":1}
Sep 24 01:03:30 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:30 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:30.970377 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:30.960464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:33705)
0924 01:03:30.961099 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:30.961116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:30.961294 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:30.961841 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:30.961847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:30.962434 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:30.962625 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:30.962635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.963634 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.963637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:30.964541 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:30.964551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.965233 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.965237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:30.965351 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:30.966357 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:30.966379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:30.970008 (+ 3629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":298,"threads_started":1}
W20260924 01:03:30.970413 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:30.960668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:33705 (local address 127.8.254.126:44031)
0924 01:03:30.960900 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:30.960906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:30.961217 (+ 311us) server_negotiation.cc:407] Connection header received
0924 01:03:30.961552 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:30.961556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:30.961622 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:30.961771 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:30.962762 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.963499 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.964672 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:30.965075 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:30.965211 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:30.970230 (+ 5019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:33705: BlockingRecv error: recv got EOF from 127.8.254.65:33705 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 01:03:31 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:31 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:31.312804 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:31.302993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:33721)
0924 01:03:31.303565 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:31.303588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:31.303702 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:31.304020 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:31.304023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:31.304427 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:31.304626 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:31.304636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.306113 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.306118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:31.307136 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:31.307151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.307946 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.307951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:31.308073 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:31.308700 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:31.308717 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:31.312562 (+ 3845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":325,"threads_started":1}
W20260924 01:03:31.313117 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:31.303116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:33721 (local address 127.8.254.126:44031)
0924 01:03:31.303290 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:31.303294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:31.303668 (+ 374us) server_negotiation.cc:407] Connection header received
0924 01:03:31.303819 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:31.303823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:31.303887 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:31.303971 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:31.304856 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.305916 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.307308 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.307754 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.307854 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:31.312831 (+ 4977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:33721: BlockingRecv error: recv got EOF from 127.8.254.66:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 01:03:31 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:31 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:31.852556 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:31.839314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:35223)
0924 01:03:31.839871 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:31.839888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:31.839994 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:31.840501 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:31.840504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:31.840894 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:31.841088 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:31.841099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.842183 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.842187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:31.843180 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:31.843193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.843886 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.843891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:31.844005 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:31.848694 (+ 4689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:31.848713 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:31.852175 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":268,"threads_started":1}
W20260924 01:03:31.852741 18576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:31.839498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:35223 (local address 127.8.254.126:44031)
0924 01:03:31.840194 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:31.840199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:31.840214 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:03:31.840259 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:31.840263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:31.840330 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:31.840418 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:31.841277 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.842058 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.843308 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.843731 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.843822 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:31.852419 (+ 8597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:35223: BlockingRecv error: recv got EOF from 127.8.254.67:35223 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":464,"threads_started":1}
Sep 24 01:03:31 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:31 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:31.981256 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:31.971555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:57593)
0924 01:03:31.972134 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:31.972178 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:31.972313 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:31.972654 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:31.972658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:31.973231 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:31.973423 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:31.973434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.974544 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.974548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:31.975512 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:31.975525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.976308 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.976313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:31.976441 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:31.977111 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:31.977130 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:31.980911 (+ 3781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
W20260924 01:03:31.981339 18576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:31.971686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:57593 (local address 127.8.254.126:44031)
0924 01:03:31.971937 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:31.971943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:31.972284 (+ 341us) server_negotiation.cc:407] Connection header received
0924 01:03:31.972418 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:31.972422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:31.972487 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:31.972569 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:31.973562 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.974409 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.975652 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:31.976052 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:31.976250 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:31.981118 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:57593: BlockingRecv error: recv got EOF from 127.8.254.65:57593 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 24 01:03:32 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:32 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:32.323292 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:32.313891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:45145)
0924 01:03:32.314464 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:32.314480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:32.314591 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:32.314932 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:32.314935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:32.315380 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:32.315535 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:32.315545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.316547 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.316551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:32.317484 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:32.317496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.318372 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.318379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:32.318592 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:32.319336 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:32.319354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:32.322943 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":319,"threads_started":1}
W20260924 01:03:32.323436 18576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:32.313924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:45145 (local address 127.8.254.126:44031)
0924 01:03:32.314149 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:32.314153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:32.314573 (+ 420us) server_negotiation.cc:407] Connection header received
0924 01:03:32.314703 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:32.314707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:32.314802 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:32.314889 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:32.315677 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.316424 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.317616 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.318156 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.318313 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:32.323127 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:45145: BlockingRecv error: recv got EOF from 127.8.254.66:45145 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 24 01:03:32 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:32 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:32.863216 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:32.853681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:41559)
0924 01:03:32.854244 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:32.854260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:32.854370 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:32.854734 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:32.854738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:32.855167 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:32.855311 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:32.855320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.856301 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.856305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:32.857251 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:32.857263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.857951 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.857974 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:32.858100 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:32.858712 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:32.858732 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:32.862986 (+ 4254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":318,"threads_started":1}
W20260924 01:03:32.863469 18580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:32.853702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:41559 (local address 127.8.254.126:44031)
0924 01:03:32.854169 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:32.854173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:32.854341 (+ 168us) server_negotiation.cc:407] Connection header received
0924 01:03:32.854488 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:32.854492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:32.854562 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:32.854669 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:32.855441 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.856172 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.857378 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.857812 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.857904 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:32.863193 (+ 5289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:41559: BlockingRecv error: recv got EOF from 127.8.254.67:41559 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":264,"threads_started":1}
Sep 24 01:03:32 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:32 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:32.991859 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:32.982345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:49957)
0924 01:03:32.982837 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:32.982877 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:32.983000 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:32.983319 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:32.983323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:32.983674 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:32.983872 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:32.983883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.985355 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.985362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:32.986549 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:32.986563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.987283 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.987289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:32.987465 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:32.988062 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:32.988079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:32.991521 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":268,"threads_started":1}
W20260924 01:03:32.991997 18580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:32.982463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:49957 (local address 127.8.254.126:44031)
0924 01:03:32.982718 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:32.982724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:32.982964 (+ 240us) server_negotiation.cc:407] Connection header received
0924 01:03:32.983109 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:32.983113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:32.983176 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:32.983261 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:32.984068 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.985212 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.986705 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:32.987116 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:32.987259 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:32.991742 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:49957: BlockingRecv error: recv got EOF from 127.8.254.65:49957 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 24 01:03:33 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:33 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:33.334542 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:33.324422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:44237)
0924 01:03:33.324984 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:33.325002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:33.325178 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:33.325564 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:33.325569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:33.325963 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:33.326133 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:33.326143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.327262 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.327269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:33.328271 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:33.328284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.329242 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.329249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:33.329452 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:33.330185 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:33.330281 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:33.334144 (+ 3863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":318,"threads_started":1}
W20260924 01:03:33.334631 18580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:33.324513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:44237 (local address 127.8.254.126:44031)
0924 01:03:33.324769 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:33.324777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:33.325105 (+ 328us) server_negotiation.cc:407] Connection header received
0924 01:03:33.325317 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:33.325322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:33.325406 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:33.325523 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:33.326320 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.327103 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.328431 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.328927 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.329057 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:33.334419 (+ 5362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:44237: BlockingRecv error: recv got EOF from 127.8.254.66:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 01:03:33 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:33 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:33.886026 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:33.864495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:50945)
0924 01:03:33.865101 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:33.865122 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:33.865260 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:33.865901 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:33.865905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:33.866483 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:33.866670 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:33.866680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.867736 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.867740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:33.868709 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:33.868720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.869483 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.869487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:33.869611 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:33.881756 (+ 12145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:33.881780 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:33.885751 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":340,"threads_started":1}
W20260924 01:03:33.886420 18584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:33.864818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:50945 (local address 127.8.254.126:44031)
0924 01:03:33.865574 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:33.865578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:33.865594 (+ 16us) server_negotiation.cc:407] Connection header received
0924 01:03:33.865640 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:33.865644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:33.865726 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:33.865869 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:33.866815 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.867615 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.868845 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.869338 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.869436 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:33.886109 (+ 16673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:50945: BlockingRecv error: recv got EOF from 127.8.254.67:50945 (error 108)
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":536,"threads_started":1}
Sep 24 01:03:33 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:33 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:34.002943 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:33.993009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:39437)
0924 01:03:33.993582 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:33.993598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:33.993769 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:33.994325 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:33.994330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:33.994775 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:33.994960 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:33.994971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.996029 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.996035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:33.996984 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:33.996996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.997819 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.997825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:33.998006 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:33.998727 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:33.998746 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:34.002600 (+ 3854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
W20260924 01:03:34.003018 18584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:33.993110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:39437 (local address 127.8.254.126:44031)
0924 01:03:33.993381 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:33.993389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:33.993695 (+ 306us) server_negotiation.cc:407] Connection header received
0924 01:03:33.993938 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:33.993942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:33.994125 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:33.994295 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:33.995121 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.995868 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.997159 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:33.997650 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:33.997796 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:34.002810 (+ 5014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:39437: BlockingRecv error: recv got EOF from 127.8.254.65:39437 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 01:03:34 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:34 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:34.345292 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:34.335629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:55273)
0924 01:03:34.336210 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:34.336229 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:34.336392 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:34.336761 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:34.336766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:34.337226 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:34.337405 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:34.337415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.338577 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.338583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:34.339532 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:34.339547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.340319 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.340325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:34.340523 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:34.341189 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:34.341208 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:34.345051 (+ 3843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
W20260924 01:03:34.345572 18584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:34.335699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:55273 (local address 127.8.254.126:44031)
0924 01:03:34.335916 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:34.335925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:34.336325 (+ 400us) server_negotiation.cc:407] Connection header received
0924 01:03:34.336530 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:34.336535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:34.336617 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:34.336728 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:34.337571 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.338432 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.339705 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.340130 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.340291 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:34.345258 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:55273: BlockingRecv error: recv got EOF from 127.8.254.66:55273 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 01:03:34 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:34 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:34.896930 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:34.887159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:56353)
0924 01:03:34.887684 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:34.887707 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:34.887811 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:34.888467 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:34.888471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:34.889016 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:34.889286 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:34.889296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.890522 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.890528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:34.891451 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:34.891463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.892196 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.892202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:34.892335 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:34.892998 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:34.893016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:34.896710 (+ 3694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":290,"threads_started":1}
W20260924 01:03:34.897522 18588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:34.887308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:56353 (local address 127.8.254.126:44031)
0924 01:03:34.888084 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:34.888092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:34.888112 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:03:34.888182 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:34.888187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:34.888274 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:34.888399 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:34.889499 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.890369 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.891581 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:34.892016 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:34.892109 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:34.896979 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:56353: BlockingRecv error: recv got EOF from 127.8.254.67:56353 (error 108)
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":260,"threads_started":1}
Sep 24 01:03:35 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:35 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:35.013578 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:35.004071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:45723)
0924 01:03:35.004567 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:35.004583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:35.004701 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:35.005038 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:35.005042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:35.005426 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:35.005605 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:35.005614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.006785 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.006789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:35.007724 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:35.007736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.008374 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.008378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:35.008511 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:35.009181 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:35.009206 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:35.013172 (+ 3966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":268,"threads_started":1}
W20260924 01:03:35.013665 18588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:35.004260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:45723 (local address 127.8.254.126:44031)
0924 01:03:35.004492 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:35.004499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:35.004666 (+ 167us) server_negotiation.cc:407] Connection header received
0924 01:03:35.004810 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:35.004814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:35.004878 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:35.004962 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:35.005737 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.006637 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.007856 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.008246 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.008338 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:35.013399 (+ 5061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:45723: BlockingRecv error: recv got EOF from 127.8.254.65:45723 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 01:03:35 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:35 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:35.356330 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:35.346430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:54867)
0924 01:03:35.347072 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:35.347094 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:35.347245 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:35.347666 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:35.347671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:35.348167 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:35.348344 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:35.348354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.349453 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.349460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:35.350485 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:35.350500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.351301 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.351307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:35.351495 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:35.352109 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:35.352127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:35.355979 (+ 3852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":332,"threads_started":1}
W20260924 01:03:35.356555 18588 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:35.346416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54867 (local address 127.8.254.126:44031)
0924 01:03:35.346733 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:35.346743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:35.347221 (+ 478us) server_negotiation.cc:407] Connection header received
0924 01:03:35.347375 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:35.347380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:35.347487 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:35.347625 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:35.348524 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.349296 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.350653 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.351102 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.351252 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:35.356204 (+ 4952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:54867: BlockingRecv error: recv got EOF from 127.8.254.66:54867 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 01:03:35 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:35 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:35.907505 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:35.897938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:33133)
0924 01:03:35.898552 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:35.898579 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:35.898738 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:35.899161 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:35.899164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:35.899680 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:35.899846 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:35.899855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.901092 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.901098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:35.902114 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:35.902127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.902944 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.902950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:35.903150 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:35.903777 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:35.903794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:35.907258 (+ 3464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":359,"threads_started":1}
W20260924 01:03:35.907979 18592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:35.898114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:33133 (local address 127.8.254.126:44031)
0924 01:03:35.898853 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:35.898858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:35.898876 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:03:35.898929 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:35.898933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:35.899001 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:35.899108 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:35.900025 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.900930 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.902332 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:35.902755 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:35.902899 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:35.907567 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:33133: BlockingRecv error: recv got EOF from 127.8.254.67:33133 (error 108)
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":503,"threads_started":1}
Sep 24 01:03:36 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:36 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 01:03:36.024067 18592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:36.014676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:52527 (local address 127.8.254.126:44031)
0924 01:03:36.014875 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:36.014882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:36.015192 (+ 310us) server_negotiation.cc:407] Connection header received
0924 01:03:36.015397 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:36.015403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:36.015479 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:36.015594 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:36.016417 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.017177 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.018507 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.018932 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.019082 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:36.023833 (+ 4751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:52527: BlockingRecv error: recv got EOF from 127.8.254.65:52527 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 01:03:36.024057 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:36.014557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:52527)
0924 01:03:36.015089 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:36.015105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:36.015267 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:36.015646 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:36.015651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:36.016081 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:36.016252 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:36.016261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.017345 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.017351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:36.018360 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:36.018372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.019104 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.019110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:36.019297 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:36.019890 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:36.019908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:36.023673 (+ 3765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":303,"threads_started":1}
Sep 24 01:03:36 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:36 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:36.366659 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:36.357345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:58225)
0924 01:03:36.357918 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:36.357935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:36.358071 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:36.358477 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:36.358480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:36.358940 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:36.359117 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:36.359126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.360136 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.360140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:36.361188 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:36.361203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.361948 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.361952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:36.362086 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:36.362816 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:36.362834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:36.366301 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":357,"threads_started":1}
W20260924 01:03:36.366834 18592 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:36.357619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:58225 (local address 127.8.254.126:44031)
0924 01:03:36.357801 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:36.357805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:36.358012 (+ 207us) server_negotiation.cc:407] Connection header received
0924 01:03:36.358171 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:36.358175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:36.358345 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:36.358431 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:36.359245 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.359983 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.361323 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.361805 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.361901 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:36.366522 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:58225: BlockingRecv error: recv got EOF from 127.8.254.66:58225 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 01:03:36 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:36 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.67@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:36.918610 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:36.908644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.67:37845)
0924 01:03:36.909184 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:36.909202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:36.909316 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:36.909860 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:36.909867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:36.910571 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:36.910817 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:36.910831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.911903 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.911907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:36.912908 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:36.912920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.913599 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.913604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:36.913781 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:36.914671 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:36.914691 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:36.918359 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
W20260924 01:03:36.918964 18596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:36.908758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:37845 (local address 127.8.254.126:44031)
0924 01:03:36.909491 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:36.909497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:36.909513 (+ 16us) server_negotiation.cc:407] Connection header received
0924 01:03:36.909566 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:36.909572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:36.909666 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:36.909793 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:36.910983 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.911769 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.913050 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:36.913480 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:36.913570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:36.918556 (+ 4986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.67:37845: BlockingRecv error: recv got EOF from 127.8.254.67:37845 (error 108)
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":521,"threads_started":1}
Sep 24 01:03:37 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:37 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.65@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:37.034379 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:37.025150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.65:60301)
0924 01:03:37.025633 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:37.025649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:37.025776 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:37.026145 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:37.026148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:37.026591 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:37.026761 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:37.026769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.027839 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.027843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:37.028773 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:37.028783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.029447 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.029451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:37.029579 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.030207 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:37.030226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:37.034005 (+ 3779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
W20260924 01:03:37.034482 18596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:37.025303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:60301 (local address 127.8.254.126:44031)
0924 01:03:37.025656 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:37.025660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:37.025704 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:03:37.025926 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:37.025931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:37.025997 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:37.026085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:37.026905 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.027692 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.028915 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.029309 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.029414 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.034233 (+ 4819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.65:60301: BlockingRecv error: recv got EOF from 127.8.254.65:60301 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
Sep 24 01:03:37 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 01:03:37 dist-test-slave-5c7l krb5kdc[17534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.254.66@KRBTEST.COM for oryx/127.8.254.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 01:03:37.377804 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:37.367973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44031 (local address 127.8.254.66:48255)
0924 01:03:37.368528 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:37.368544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:37.368660 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:37.368986 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:37.368989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:37.369584 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:37.369885 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:37.369900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.371139 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.371143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:37.372051 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:37.372063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.372733 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.372737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:37.372862 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.373454 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:37.373472 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:37.377463 (+ 3991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.254.126:44031: Server oryx/127.8.254.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
W20260924 01:03:37.377998 18596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:37.368057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:48255 (local address 127.8.254.126:44031)
0924 01:03:37.368423 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:37.368428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:37.368635 (+ 207us) server_negotiation.cc:407] Connection header received
0924 01:03:37.368768 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:37.368772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:37.368851 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:37.368938 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:37.370068 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.370994 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.372195 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.372601 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.372693 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.377672 (+ 4979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.254.66:48255: BlockingRecv error: recv got EOF from 127.8.254.66:48255 (error 108)
Metrics: {"server-negotiator.queue_time_us":247}
I20260924 01:03:37.486446 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18123
I20260924 01:03:37.501843 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:37.488076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.8.254.65:41909)
0924 01:03:37.488997 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:37.489008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:37.489039 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:37.489115 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:37.489122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:37.489365 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:37.489574 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:37.491696 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.493421 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.494977 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.495587 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.495691 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.498363 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:37.498371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:37.498385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:37.498434 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:37.501154 (+ 2720us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.254.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 01:03:37.501453 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32816: Request ticket server kudu/127.8.254.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":543,"threads_started":1}
W20260924 01:03:37.501950 18600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32816: Request ticket server kudu/127.8.254.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 01:03:37.501971 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18123: Not authorized: Client connection negotiation failed: client connection to 127.8.254.65:41909: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.254.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 01:03:37.502246 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18123
I20260924 01:03:37.520646 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18275
I20260924 01:03:37.533987 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:37.521924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59842 (local address 127.8.254.66:35087)
0924 01:03:37.522652 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:37.522659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:37.522688 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:03:37.522754 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:37.522763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:37.522998 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:37.523254 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:37.524604 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.525724 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.527096 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.527709 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.527874 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.530516 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:37.530526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:37.530541 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:37.530587 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:37.533199 (+ 2612us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.254.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 01:03:37.533621 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59842: Request ticket server kudu/127.8.254.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":399,"threads_started":1}
W20260924 01:03:37.534091 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18275: Not authorized: Client connection negotiation failed: client connection to 127.8.254.66:35087: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.254.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 01:03:37.534127 18601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59842: Request ticket server kudu/127.8.254.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 01:03:37.534323 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18275
I20260924 01:03:37.551610 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18408
I20260924 01:03:37.564713 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:37.552835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.8.254.67:42095)
0924 01:03:37.553575 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:37.553585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:37.553634 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:03:37.553760 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:37.553769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:37.554101 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:37.554329 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:37.555664 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.556848 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.558044 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:37.558721 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:37.558822 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:37.561472 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:37.561481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:37.561495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:37.561542 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:37.564105 (+ 2563us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.254.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 01:03:37.564392 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59424: Request ticket server kudu/127.8.254.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":370,"threads_started":1}
W20260924 01:03:37.564848 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18408: Not authorized: Client connection negotiation failed: client connection to 127.8.254.67:42095: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.254.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 01:03:37.564893 18602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59424: Request ticket server kudu/127.8.254.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 01:03:37.565045 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18408
I20260924 01:03:37.581904 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18054
I20260924 01:03:37.583464 18116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:37.701148 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18054
2026-09-24T01:03:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21400 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T01:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:37Z Disabled control of system clock
I20260924 01:03:37.781198 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45959
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 01:03:38.119441 18607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:38.119738 18607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:38.119817 18607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:38.127053 18607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:38.127161 18607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:38.127202 18607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:38.127260 18607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:38.127305 18607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:38.136524 18607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:38.137907 18607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:38.139667 18607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:38.152040 18615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:38.152145 18612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:38.152160 18613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:38.153038 18607 server_base.cc:1034] running on GCE node
I20260924 01:03:38.153915 18607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:38.155510 18607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:38.156773 18607 hybrid_clock.cc:648] HybridClock initialized: now 1790211818156731 us; error 57 us; skew 500 ppm
I20260924 01:03:38.157150 18607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:38.159600 18607 webserver.cc:460] Webserver started at http://127.8.254.126:34167/ using document root <none> and password file <none>
I20260924 01:03:38.160346 18607 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:38.160492 18607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:38.160833 18607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:38.163908 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0"
format_stamp: "Formatted at 2026-09-24 01:03:38 on dist-test-slave-5c7l"
server_key: "ab222100c899af80fa43063b02b76475"
server_key_iv: "65bc45caf3d5f16df07292680829f13d"
server_key_version: "encryptionkey@0"
I20260924 01:03:38.164718 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0"
format_stamp: "Formatted at 2026-09-24 01:03:38 on dist-test-slave-5c7l"
server_key: "ab222100c899af80fa43063b02b76475"
server_key_iv: "65bc45caf3d5f16df07292680829f13d"
server_key_version: "encryptionkey@0"
I20260924 01:03:38.170135 18607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 01:03:38.173723 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:38.175189 18607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:38.175390 18607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0"
format_stamp: "Formatted at 2026-09-24 01:03:38 on dist-test-slave-5c7l"
server_key: "ab222100c899af80fa43063b02b76475"
server_key_iv: "65bc45caf3d5f16df07292680829f13d"
server_key_version: "encryptionkey@0"
I20260924 01:03:38.175631 18607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:38.189908 18607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:38.190840 18607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:38.191152 18607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:38.209836 18607 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45959
I20260924 01:03:38.209842 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45959 every 8 connection(s)
I20260924 01:03:38.211288 18607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:03:38.214282 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18607
I20260924 01:03:38.214517 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:03:38.215319 9209 external_mini_cluster.cc:1428] Setting key 81080b2ae2b385aad0692c11289d4e5f
I20260924 01:03:38.215569 18673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:38.229471 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:38.217669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.8.254.126:45959)
0924 01:03:38.218696 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:38.218706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:38.218747 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:03:38.220324 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:38.220344 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:38.220791 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:38.221120 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:38.221798 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:38.223210 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:38.225024 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:38.225680 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:38.225813 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:38.227125 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:38.227133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:38.227146 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:38.227735 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:38.227815 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:38.227836 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:38.227936 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:38.228168 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 01:03:38.228522 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":378,"threads_started":1}
I20260924 01:03:38.229864 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0: Bootstrap starting.
I20260924 01:03:38.233039 18673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:38.234097 18673 log.cc:824] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:38.236992 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0: No bootstrap required, opened a new log
I20260924 01:03:38.244292 18673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45959 } }
I20260924 01:03:38.244715 18673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:38.244843 18673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47db312ee9ce4f119a1dd4d3d0bcdcc0, State: Initialized, Role: FOLLOWER
I20260924 01:03:38.245437 18673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45959 } }
I20260924 01:03:38.245685 18673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:03:38.245838 18673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:03:38.246019 18673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:38.248277 18673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45959 } }
I20260924 01:03:38.248878 18673 leader_election.cc:302] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47db312ee9ce4f119a1dd4d3d0bcdcc0; no voters:
I20260924 01:03:38.249565 18673 leader_election.cc:288] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:03:38.249774 18678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:38.251135 18678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [term 1 LEADER]: Becoming Leader. State: Replica: 47db312ee9ce4f119a1dd4d3d0bcdcc0, State: Running, Role: LEADER
I20260924 01:03:38.251673 18678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45959 } }
I20260924 01:03:38.252161 18673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:38.254732 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45959 } } }
I20260924 01:03:38.255139 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:38.254707 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47db312ee9ce4f119a1dd4d3d0bcdcc0. Latest consensus state: current_term: 1 leader_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47db312ee9ce4f119a1dd4d3d0bcdcc0" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45959 } } }
I20260924 01:03:38.255661 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:38.257134 18685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:38.263612 18685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:38.274420 18685 catalog_manager.cc:1347] Generated new cluster ID: 2c01171775e246fb969dae10e4768afc
I20260924 01:03:38.274540 18685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:38.286561 18685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:03:38.288448 18685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:38.299746 18685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47db312ee9ce4f119a1dd4d3d0bcdcc0: Generated new TSK 0
I20260924 01:03:38.300803 18685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:03:38.313467 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45959
--builtin_ntp_servers=127.8.254.84:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 01:03:38.649124 18697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:38.649514 18697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:38.649669 18697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:38.658083 18697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:38.658244 18697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:38.658428 18697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:38.668159 18697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:38.669565 18697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:38.671504 18697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:38.685467 18702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:38.685489 18703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:38.685635 18705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:38.691720 18697 server_base.cc:1034] running on GCE node
I20260924 01:03:38.692505 18697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:38.693614 18697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:38.694854 18697 hybrid_clock.cc:648] HybridClock initialized: now 1790211818694797 us; error 51 us; skew 500 ppm
I20260924 01:03:38.695204 18697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:38.697533 18697 webserver.cc:460] Webserver started at http://127.8.254.65:46457/ using document root <none> and password file <none>
I20260924 01:03:38.698362 18697 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:38.698503 18697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:38.698855 18697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:38.701833 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "7ea87029cb814ab99fe2d18468ca6d89"
format_stamp: "Formatted at 2026-09-24 01:03:38 on dist-test-slave-5c7l"
server_key: "daf21acd21d486afdff20dfdeca91b40"
server_key_iv: "aa30bd71ac5b4f8f75b5fef1c3d1489b"
server_key_version: "encryptionkey@0"
I20260924 01:03:38.702701 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "7ea87029cb814ab99fe2d18468ca6d89"
format_stamp: "Formatted at 2026-09-24 01:03:38 on dist-test-slave-5c7l"
server_key: "daf21acd21d486afdff20dfdeca91b40"
server_key_iv: "aa30bd71ac5b4f8f75b5fef1c3d1489b"
server_key_version: "encryptionkey@0"
I20260924 01:03:38.708261 18697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:03:38.712921 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:38.714422 18697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:38.714614 18697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "7ea87029cb814ab99fe2d18468ca6d89"
format_stamp: "Formatted at 2026-09-24 01:03:38 on dist-test-slave-5c7l"
server_key: "daf21acd21d486afdff20dfdeca91b40"
server_key_iv: "aa30bd71ac5b4f8f75b5fef1c3d1489b"
server_key_version: "encryptionkey@0"
I20260924 01:03:38.714852 18697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:38.727149 18697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:38.727946 18697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:38.728204 18697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:38.729266 18697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:38.730738 18697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:38.730829 18697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:38.730916 18697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:38.730994 18697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:38.764892 18697 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:46417
I20260924 01:03:38.764943 18823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:46417 every 8 connection(s)
I20260924 01:03:38.766362 18697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:38.776587 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18697
I20260924 01:03:38.776831 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:03:38.777628 9209 external_mini_cluster.cc:1428] Setting key f0d830e70bfeac85f5d827d7c683316a
I20260924 01:03:38.779518 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:38.768965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:49663 (local address 127.8.254.126:45959)
0924 01:03:38.769551 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:38.769558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:38.770816 (+ 1258us) server_negotiation.cc:407] Connection header received
0924 01:03:38.772664 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:38.772668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:38.772739 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:38.772837 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:38.773929 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:38.774712 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:38.776290 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:38.776975 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:38.777120 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:38.778517 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:38.778522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:38.778526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:38.778544 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:38.778615 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:38.778632 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:38.778728 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:38.778995 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 01:03:38.779135 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
I20260924 01:03:38.780726 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:38.769676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45959 (local address 127.8.254.65:49663)
0924 01:03:38.770648 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:38.770691 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:38.772400 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:38.773038 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:38.773047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:38.773148 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:38.773771 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:38.773789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:38.774852 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:38.774856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:38.776142 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:38.776154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:38.777189 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:38.777194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:38.777392 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:38.778098 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:03:38.778101 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:38.778149 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:03:38.778296 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:38.778740 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:38.778749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:38.778755 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 01:03:38.778969 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 01:03:38.779396 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":271,"threads_started":1}
I20260924 01:03:38.780727 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45959
--builtin_ntp_servers=127.8.254.84:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 01:03:38.782526 18824 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45959
I20260924 01:03:38.782855 18824 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:38.783803 18824 heartbeater.cc:511] Master 127.8.254.126:45959 requested a full tablet report, sending...
I20260924 01:03:38.786299 18638 ts_manager.cc:194] Registered new tserver with Master: 7ea87029cb814ab99fe2d18468ca6d89 (127.8.254.65:46417)
I20260924 01:03:38.788556 18638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.65:49663
W20260924 01:03:39.115124 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:39.115592 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:39.115733 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:39.126147 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:39.126319 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:39.126500 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:39.136270 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:39.137636 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:39.139506 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:39.153376 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:39.153611 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:39.153306 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:39.154685 18828 server_base.cc:1034] running on GCE node
I20260924 01:03:39.155349 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:39.156363 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:39.157604 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790211819157565 us; error 41 us; skew 500 ppm
I20260924 01:03:39.157963 18828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:39.167094 18828 webserver.cc:460] Webserver started at http://127.8.254.66:40801/ using document root <none> and password file <none>
I20260924 01:03:39.167908 18828 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:39.168059 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:39.168416 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:39.171502 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "66f7d3fcdf974bd3bbb9d3d1f50ff902"
format_stamp: "Formatted at 2026-09-24 01:03:39 on dist-test-slave-5c7l"
server_key: "e0e56a43eeccc334c1312f69a2881f15"
server_key_iv: "58489a333300fa0facc039712c9e10a5"
server_key_version: "encryptionkey@0"
I20260924 01:03:39.172317 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "66f7d3fcdf974bd3bbb9d3d1f50ff902"
format_stamp: "Formatted at 2026-09-24 01:03:39 on dist-test-slave-5c7l"
server_key: "e0e56a43eeccc334c1312f69a2881f15"
server_key_iv: "58489a333300fa0facc039712c9e10a5"
server_key_version: "encryptionkey@0"
I20260924 01:03:39.177611 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:39.181398 18842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:39.182857 18828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:39.183053 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "66f7d3fcdf974bd3bbb9d3d1f50ff902"
format_stamp: "Formatted at 2026-09-24 01:03:39 on dist-test-slave-5c7l"
server_key: "e0e56a43eeccc334c1312f69a2881f15"
server_key_iv: "58489a333300fa0facc039712c9e10a5"
server_key_version: "encryptionkey@0"
I20260924 01:03:39.183284 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:39.216280 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:39.217197 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:39.217521 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:39.218652 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:39.220280 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:39.220367 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:39.220532 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:39.220603 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:39.255420 18828 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:33355
I20260924 01:03:39.255470 18954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:33355 every 8 connection(s)
I20260924 01:03:39.256789 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:39.266531 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18828
I20260924 01:03:39.266748 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:03:39.267168 9209 external_mini_cluster.cc:1428] Setting key cacf4069c4e6e91eeb1b054388a2353f
I20260924 01:03:39.269948 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45959
--builtin_ntp_servers=127.8.254.84:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 01:03:39.270182 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.259452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:57139 (local address 127.8.254.126:45959)
0924 01:03:39.259663 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:39.259667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:39.261098 (+ 1431us) server_negotiation.cc:407] Connection header received
0924 01:03:39.263008 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:39.263014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:39.263108 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:39.263249 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:39.264285 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.265023 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.266623 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.267157 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.267310 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.269384 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:39.269389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:39.269392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:39.269410 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:39.269442 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:39.269454 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:39.269551 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:39.269779 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 01:03:39.269875 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 01:03:39.271262 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.259999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45959 (local address 127.8.254.66:57139)
0924 01:03:39.260929 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:39.260966 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:39.262726 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:39.263464 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:39.263474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:39.263564 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:39.264118 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:39.264134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.265171 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.265177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:39.266475 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:39.266487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.267386 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.267393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:39.267624 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.269002 (+ 1378us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:03:39.269006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:39.269061 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:03:39.269164 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:39.269561 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:39.269573 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:39.269581 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 01:03:39.269769 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 01:03:39.270189 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":269,"threads_started":1}
I20260924 01:03:39.272697 18955 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45959
I20260924 01:03:39.273007 18955 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:39.273787 18955 heartbeater.cc:511] Master 127.8.254.126:45959 requested a full tablet report, sending...
I20260924 01:03:39.275204 18638 ts_manager.cc:194] Registered new tserver with Master: 66f7d3fcdf974bd3bbb9d3d1f50ff902 (127.8.254.66:33355)
I20260924 01:03:39.276012 18638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.66:57139
W20260924 01:03:39.605962 18958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:39.606338 18958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:39.606457 18958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:39.613837 18958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:39.613947 18958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:39.614053 18958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:39.623643 18958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:39.624909 18958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:39.626701 18958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:39.640426 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:39.640514 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:39.640451 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:39.641341 18958 server_base.cc:1034] running on GCE node
I20260924 01:03:39.641968 18958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:39.643007 18958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:39.644240 18958 hybrid_clock.cc:648] HybridClock initialized: now 1790211819644194 us; error 31 us; skew 500 ppm
I20260924 01:03:39.644577 18958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:39.647135 18958 webserver.cc:460] Webserver started at http://127.8.254.67:34405/ using document root <none> and password file <none>
I20260924 01:03:39.647866 18958 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:39.647964 18958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:39.648321 18958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:39.651387 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "d690a89c793c48528f4db33fe18d1d5b"
format_stamp: "Formatted at 2026-09-24 01:03:39 on dist-test-slave-5c7l"
server_key: "f2ef6844fbc521bccaad8ad4db5282f0"
server_key_iv: "17b16ce6cc03198f6fd6096740090dd9"
server_key_version: "encryptionkey@0"
I20260924 01:03:39.652251 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "d690a89c793c48528f4db33fe18d1d5b"
format_stamp: "Formatted at 2026-09-24 01:03:39 on dist-test-slave-5c7l"
server_key: "f2ef6844fbc521bccaad8ad4db5282f0"
server_key_iv: "17b16ce6cc03198f6fd6096740090dd9"
server_key_version: "encryptionkey@0"
I20260924 01:03:39.657763 18958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 01:03:39.661454 18972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:39.663054 18958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 01:03:39.663296 18958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "d690a89c793c48528f4db33fe18d1d5b"
format_stamp: "Formatted at 2026-09-24 01:03:39 on dist-test-slave-5c7l"
server_key: "f2ef6844fbc521bccaad8ad4db5282f0"
server_key_iv: "17b16ce6cc03198f6fd6096740090dd9"
server_key_version: "encryptionkey@0"
I20260924 01:03:39.663546 18958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:39.694130 18958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:39.695127 18958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:39.695452 18958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:39.696478 18958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:39.698017 18958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:39.698154 18958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:39.698320 18958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:39.698416 18958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:39.733294 18958 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:45447
I20260924 01:03:39.733438 19084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:45447 every 8 connection(s)
I20260924 01:03:39.734781 18958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:39.744030 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 18958
I20260924 01:03:39.744223 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:03:39.744722 9209 external_mini_cluster.cc:1428] Setting key d8c5426ed1ef0b96e087a0fef178a8da
I20260924 01:03:39.747916 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.737432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:41509 (local address 127.8.254.126:45959)
0924 01:03:39.737644 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:39.737648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:39.739046 (+ 1398us) server_negotiation.cc:407] Connection header received
0924 01:03:39.740794 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:39.740798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:39.740874 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:39.740973 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:39.742083 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.742929 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.744650 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.745306 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.745407 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.746960 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:39.746964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:39.746968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:39.747028 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:39.747067 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:39.747083 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:39.747189 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:39.747475 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 01:03:39.747592 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 01:03:39.749217 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.737936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45959 (local address 127.8.254.67:41509)
0924 01:03:39.738890 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:39.738928 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:39.740503 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:39.741203 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:39.741212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:39.741304 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:39.741904 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:39.741921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.743088 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.743092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:39.744500 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:39.744514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.745552 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.745567 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:39.745850 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.746619 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:03:39.746622 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:39.746664 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:03:39.746747 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:39.747194 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:39.747203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:39.747230 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 01:03:39.747442 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 01:03:39.747875 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":320,"threads_started":1}
I20260924 01:03:39.750905 19085 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45959
I20260924 01:03:39.751226 19085 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:39.751994 19085 heartbeater.cc:511] Master 127.8.254.126:45959 requested a full tablet report, sending...
I20260924 01:03:39.753453 18638 ts_manager.cc:194] Registered new tserver with Master: d690a89c793c48528f4db33fe18d1d5b (127.8.254.67:45447)
I20260924 01:03:39.754365 18638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.67:41509
I20260924 01:03:39.760919 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 01:03:39.767643 18826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:39.765895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.8.254.126:45959)
0924 01:03:39.766102 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:39.766111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:39.766568 (+ 457us) server_negotiation.cc:407] Connection header received
0924 01:03:39.766746 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:39.766750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:39.766825 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:39.766935 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:39.767337 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50826: BlockingRecv error: recv got EOF from 127.0.0.1:50826 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 01:03:39.768585 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18697
I20260924 01:03:39.776855 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.769598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.8.254.65:46417)
0924 01:03:39.770217 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:39.770226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:39.770257 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:39.770320 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:39.770328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:39.770599 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:39.770771 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:39.771139 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.772386 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.773725 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.774388 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.774489 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.775549 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:39.775558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:39.775571 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:39.776133 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:39.776180 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:39.776200 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:39.776298 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:39.776455 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 01:03:39.776564 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":284,"threads_started":1}
I20260924 01:03:39.778136 18819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:39.859596 18824 heartbeater.cc:503] Master 127.8.254.126:45959 was elected leader, sending a full tablet report...
I20260924 01:03:39.860800 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18697
W20260924 01:03:39.877931 18624 connection.cc:535] server connection from 127.8.254.65:49663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 01:03:39.878294 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18828
I20260924 01:03:39.886312 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.879161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.8.254.66:33355)
0924 01:03:39.879720 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:39.879727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:39.879753 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:03:39.879811 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:39.879817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:39.880038 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:39.880260 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:39.880659 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.881619 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.882937 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.883501 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.883652 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.884738 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:39.884746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:39.884759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:39.885363 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:39.885419 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:39.885439 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:39.885589 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:39.885747 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 01:03:39.885888 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":252,"threads_started":1}
I20260924 01:03:39.887717 18950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:39.970467 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18828
I20260924 01:03:39.987576 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18958
I20260924 01:03:39.996136 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:39.988526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.8.254.67:45447)
0924 01:03:39.989083 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:39.989090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:39.989124 (+ 34us) server_negotiation.cc:407] Connection header received
0924 01:03:39.989182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:39.989189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:39.989410 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:39.989592 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:39.990071 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.991150 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.992440 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:39.993033 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:39.993197 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:39.994635 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:39.994645 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:39.994659 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:39.995268 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:39.995318 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:39.995340 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:39.995495 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:39.995644 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 01:03:39.995783 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":251,"threads_started":1}
I20260924 01:03:39.997464 19080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:40.079334 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18958
I20260924 01:03:40.097190 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 18607
I20260924 01:03:40.098474 18668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:40.200779 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 18607
2026-09-24T01:03:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2493 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T01:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:40Z Disabled control of system clock
I20260924 01:03:40.273993 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:36427
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:36427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 01:03:40.605351 19104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:40.605648 19104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:40.605839 19104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:40.613313 19104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:40.613446 19104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:40.613550 19104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:40.613662 19104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:40.613737 19104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:40.623103 19104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:36427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:36427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:40.624532 19104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:40.626392 19104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:40.638934 19109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:40.638861 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:40.638899 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:40.639537 19104 server_base.cc:1034] running on GCE node
I20260924 01:03:40.640364 19104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:40.641918 19104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:40.643182 19104 hybrid_clock.cc:648] HybridClock initialized: now 1790211820643110 us; error 58 us; skew 500 ppm
I20260924 01:03:40.643618 19104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:40.646131 19104 webserver.cc:460] Webserver started at http://127.8.254.126:34015/ using document root <none> and password file <none>
I20260924 01:03:40.646907 19104 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:40.647029 19104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:40.647377 19104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:40.650697 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "88e70995851848e389ace4718dfa2941"
format_stamp: "Formatted at 2026-09-24 01:03:40 on dist-test-slave-5c7l"
server_key: "1130daf8e2af3e53cbb0d8883176f9d5"
server_key_iv: "43453c2f9e8c46e992e84ad487565663"
server_key_version: "encryptionkey@0"
I20260924 01:03:40.651561 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "88e70995851848e389ace4718dfa2941"
format_stamp: "Formatted at 2026-09-24 01:03:40 on dist-test-slave-5c7l"
server_key: "1130daf8e2af3e53cbb0d8883176f9d5"
server_key_iv: "43453c2f9e8c46e992e84ad487565663"
server_key_version: "encryptionkey@0"
I20260924 01:03:40.657191 19104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:40.660833 19118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:40.662366 19104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:40.662560 19104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "88e70995851848e389ace4718dfa2941"
format_stamp: "Formatted at 2026-09-24 01:03:40 on dist-test-slave-5c7l"
server_key: "1130daf8e2af3e53cbb0d8883176f9d5"
server_key_iv: "43453c2f9e8c46e992e84ad487565663"
server_key_version: "encryptionkey@0"
I20260924 01:03:40.662799 19104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:40.668946 19104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:40.669763 19104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:40.670077 19104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:40.689059 19104 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:36427
I20260924 01:03:40.689067 19169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:36427 every 8 connection(s)
I20260924 01:03:40.690586 19104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:03:40.694591 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:40.696000 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19104
I20260924 01:03:40.696202 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:03:40.696870 9209 external_mini_cluster.cc:1428] Setting key 3b1af0d2c8851479e19af2a21b5cd3ff
I20260924 01:03:40.705639 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:40.698990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.8.254.126:36427)
0924 01:03:40.700088 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:40.700103 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:40.700146 (+ 43us) server_negotiation.cc:407] Connection header received
0924 01:03:40.701770 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:40.701793 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:40.702305 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:40.702723 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:40.703864 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:40.703869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:40.703886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:40.703900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:40.703963 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:40.703979 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:40.704291 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:40.704457 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 01:03:40.704716 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":397,"threads_started":1}
I20260924 01:03:40.707056 19170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941: Bootstrap starting.
I20260924 01:03:40.710413 19170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:40.711428 19170 log.cc:824] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:40.714471 19170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941: No bootstrap required, opened a new log
I20260924 01:03:40.723487 19170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e70995851848e389ace4718dfa2941" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36427 } }
I20260924 01:03:40.723922 19170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:40.724040 19170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e70995851848e389ace4718dfa2941, State: Initialized, Role: FOLLOWER
I20260924 01:03:40.724637 19170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e70995851848e389ace4718dfa2941" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36427 } }
I20260924 01:03:40.724885 19170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:03:40.725035 19170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:03:40.725229 19170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:40.727561 19170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e70995851848e389ace4718dfa2941" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36427 } }
I20260924 01:03:40.728106 19170 leader_election.cc:302] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e70995851848e389ace4718dfa2941; no voters:
I20260924 01:03:40.728843 19170 leader_election.cc:288] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:03:40.729024 19175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:40.729780 19175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [term 1 LEADER]: Becoming Leader. State: Replica: 88e70995851848e389ace4718dfa2941, State: Running, Role: LEADER
I20260924 01:03:40.730412 19175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e70995851848e389ace4718dfa2941" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36427 } }
I20260924 01:03:40.731293 19170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:40.733795 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e70995851848e389ace4718dfa2941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e70995851848e389ace4718dfa2941" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36427 } } }
I20260924 01:03:40.734277 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:40.734956 19177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e70995851848e389ace4718dfa2941. Latest consensus state: current_term: 1 leader_uuid: "88e70995851848e389ace4718dfa2941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e70995851848e389ace4718dfa2941" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36427 } } }
I20260924 01:03:40.735280 19177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941 [sys.catalog]: This master's current role is: LEADER
W20260924 01:03:40.743775 19191 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:03:40.743945 19191 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:03:40.744117 19184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:40.746497 19184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:40.755976 19184 catalog_manager.cc:1347] Generated new cluster ID: b2dbbf0bd2054dee859263dc74b0c374
I20260924 01:03:40.756109 19184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:40.768306 19184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:03:40.768745 19184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:40.786823 19184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e70995851848e389ace4718dfa2941: Generated new TSK 0
I20260924 01:03:40.787766 19184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:03:40.801012 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:36427
--builtin_ntp_servers=127.8.254.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 01:03:41.159075 19194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:41.159384 19194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:41.159574 19194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:41.167194 19194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:41.167303 19194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:41.167505 19194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:41.177265 19194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:36427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:41.178819 19194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:41.180655 19194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:41.193913 19199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:41.193885 19200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:41.193956 19202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:41.194772 19194 server_base.cc:1034] running on GCE node
I20260924 01:03:41.195405 19194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:41.196300 19194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:41.197507 19194 hybrid_clock.cc:648] HybridClock initialized: now 1790211821197437 us; error 64 us; skew 500 ppm
I20260924 01:03:41.197856 19194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:41.200309 19194 webserver.cc:460] Webserver started at http://127.8.254.65:45337/ using document root <none> and password file <none>
I20260924 01:03:41.201088 19194 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:41.201212 19194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:41.201565 19194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:41.206096 19194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "131f4a5606b64b4ea09c01517ad77f7b"
format_stamp: "Formatted at 2026-09-24 01:03:41 on dist-test-slave-5c7l"
server_key: "14bfd46f041ef6eff7b66ff413a57690"
server_key_iv: "9823fee6a4c8f4296421be44098fc513"
server_key_version: "encryptionkey@0"
I20260924 01:03:41.207023 19194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "131f4a5606b64b4ea09c01517ad77f7b"
format_stamp: "Formatted at 2026-09-24 01:03:41 on dist-test-slave-5c7l"
server_key: "14bfd46f041ef6eff7b66ff413a57690"
server_key_iv: "9823fee6a4c8f4296421be44098fc513"
server_key_version: "encryptionkey@0"
I20260924 01:03:41.212288 19194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:03:41.215849 19208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:41.217273 19194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:03:41.217460 19194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "131f4a5606b64b4ea09c01517ad77f7b"
format_stamp: "Formatted at 2026-09-24 01:03:41 on dist-test-slave-5c7l"
server_key: "14bfd46f041ef6eff7b66ff413a57690"
server_key_iv: "9823fee6a4c8f4296421be44098fc513"
server_key_version: "encryptionkey@0"
I20260924 01:03:41.217710 19194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:41.223364 19194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:41.224167 19194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:41.224455 19194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:41.225418 19194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:41.226936 19194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:41.227073 19194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:41.227203 19194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:41.227290 19194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:41.261929 19194 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:39423
I20260924 01:03:41.262025 19320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:39423 every 8 connection(s)
I20260924 01:03:41.263362 19194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:41.268674 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19194
I20260924 01:03:41.268891 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:03:41.269682 9209 external_mini_cluster.cc:1428] Setting key 3e95fe452e34dcc5dd9c45de398f5cba
I20260924 01:03:41.272573 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:36427
--builtin_ntp_servers=127.8.254.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 01:03:41.272962 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:41.266052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:43633 (local address 127.8.254.126:36427)
0924 01:03:41.266560 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:41.266566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:41.267901 (+ 1335us) server_negotiation.cc:407] Connection header received
0924 01:03:41.270571 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:41.270578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:41.270663 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:41.270795 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:41.272184 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:41.272190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:41.272195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:41.272215 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:41.272263 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:41.272274 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:41.272409 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:41.272607 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 01:03:41.272676 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
I20260924 01:03:41.273934 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:41.266751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:36427 (local address 127.8.254.65:43633)
0924 01:03:41.267739 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:41.267782 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:41.270201 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:41.271071 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:41.271079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:41.271178 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:41.271879 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:03:41.271881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:41.271921 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:03:41.272015 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:41.272372 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:41.272384 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:41.272395 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 01:03:41.272693 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 01:03:41.272970 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":301,"threads_started":1}
I20260924 01:03:41.275527 19321 heartbeater.cc:347] Connected to a master server at 127.8.254.126:36427
I20260924 01:03:41.275862 19321 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:41.276603 19321 heartbeater.cc:511] Master 127.8.254.126:36427 requested a full tablet report, sending...
I20260924 01:03:41.278915 19135 ts_manager.cc:194] Registered new tserver with Master: 131f4a5606b64b4ea09c01517ad77f7b (127.8.254.65:39423)
W20260924 01:03:41.609856 19325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:41.610124 19325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:41.610280 19325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:41.618311 19325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:41.618410 19325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:41.618520 19325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:41.627900 19325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:36427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:41.629137 19325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:41.630949 19325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:41.644538 19330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:41.644553 19331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:41.644553 19333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:41.645387 19325 server_base.cc:1034] running on GCE node
I20260924 01:03:41.646082 19325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:41.647156 19325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:41.648408 19325 hybrid_clock.cc:648] HybridClock initialized: now 1790211821648359 us; error 48 us; skew 500 ppm
I20260924 01:03:41.648787 19325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:41.651284 19325 webserver.cc:460] Webserver started at http://127.8.254.66:45489/ using document root <none> and password file <none>
I20260924 01:03:41.652033 19325 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:41.652171 19325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:41.652515 19325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:41.655620 19325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "c9f05b80b9d140358ff48b0b8ea6670f"
format_stamp: "Formatted at 2026-09-24 01:03:41 on dist-test-slave-5c7l"
server_key: "740ee9834b91d98fe8824ba0a891f6ea"
server_key_iv: "1bd5f6a3f9538c1a401f6f5cf94042f1"
server_key_version: "encryptionkey@0"
I20260924 01:03:41.656486 19325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f05b80b9d140358ff48b0b8ea6670f"
format_stamp: "Formatted at 2026-09-24 01:03:41 on dist-test-slave-5c7l"
server_key: "740ee9834b91d98fe8824ba0a891f6ea"
server_key_iv: "1bd5f6a3f9538c1a401f6f5cf94042f1"
server_key_version: "encryptionkey@0"
I20260924 01:03:41.661938 19325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:03:41.665777 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:41.667447 19325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:41.667652 19325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "c9f05b80b9d140358ff48b0b8ea6670f"
format_stamp: "Formatted at 2026-09-24 01:03:41 on dist-test-slave-5c7l"
server_key: "740ee9834b91d98fe8824ba0a891f6ea"
server_key_iv: "1bd5f6a3f9538c1a401f6f5cf94042f1"
server_key_version: "encryptionkey@0"
I20260924 01:03:41.667905 19325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:41.673954 19325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:41.674863 19325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:41.675175 19325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:41.676203 19325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:41.677645 19325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:41.677773 19325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:41.677911 19325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:41.678001 19325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:41.714393 19325 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:35331
I20260924 01:03:41.714478 19451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:35331 every 8 connection(s)
I20260924 01:03:41.715749 19325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:41.717239 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19325
I20260924 01:03:41.717410 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:03:41.717947 9209 external_mini_cluster.cc:1428] Setting key 5e24c3a961bbf3a5c2a8618a82bbdcc0
I20260924 01:03:41.721150 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:36427
--builtin_ntp_servers=127.8.254.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 01:03:41.726157 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:41.718784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:55175 (local address 127.8.254.126:36427)
0924 01:03:41.719014 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:41.719021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:41.721391 (+ 2370us) server_negotiation.cc:407] Connection header received
0924 01:03:41.723837 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:41.723842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:41.723935 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:41.724056 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:41.725423 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:41.725435 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:41.725438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:41.725456 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:41.725487 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:41.725499 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:41.725638 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:41.725825 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 01:03:41.725897 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 01:03:41.727274 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:41.719412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:36427 (local address 127.8.254.66:55175)
0924 01:03:41.721122 (+ 1710us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:41.721179 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:41.723551 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:41.724310 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:41.724319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:41.724417 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:41.725117 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:03:41.725120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:41.725158 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:03:41.725246 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:41.725597 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:41.725609 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:41.725618 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 01:03:41.725913 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 01:03:41.726189 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":877,"threads_started":1}
I20260924 01:03:41.728848 19452 heartbeater.cc:347] Connected to a master server at 127.8.254.126:36427
I20260924 01:03:41.729162 19452 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:41.729768 19452 heartbeater.cc:511] Master 127.8.254.126:36427 requested a full tablet report, sending...
I20260924 01:03:41.731478 19135 ts_manager.cc:194] Registered new tserver with Master: c9f05b80b9d140358ff48b0b8ea6670f (127.8.254.66:35331)
W20260924 01:03:42.060585 19455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:42.060881 19455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:42.060971 19455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:42.068429 19455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:42.068534 19455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:42.068646 19455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:42.077991 19455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:36427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:42.079334 19455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:42.081019 19455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:42.094760 19460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:42.094785 19461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:42.094810 19463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:42.095847 19455 server_base.cc:1034] running on GCE node
I20260924 01:03:42.096415 19455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:42.097334 19455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:42.098531 19455 hybrid_clock.cc:648] HybridClock initialized: now 1790211822098491 us; error 35 us; skew 500 ppm
I20260924 01:03:42.098827 19455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:03:42.100988 19455 webserver.cc:460] Webserver started at http://127.8.254.67:37593/ using document root <none> and password file <none>
I20260924 01:03:42.101643 19455 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:42.101728 19455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:42.102003 19455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:42.104873 19455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "1249c0a4743c417ab23ee48a021774e5"
format_stamp: "Formatted at 2026-09-24 01:03:42 on dist-test-slave-5c7l"
server_key: "309050ae8e8029dcf684b1fd094ce18f"
server_key_iv: "4e9810b82daea2e63a2bdaacec78663d"
server_key_version: "encryptionkey@0"
I20260924 01:03:42.105707 19455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "1249c0a4743c417ab23ee48a021774e5"
format_stamp: "Formatted at 2026-09-24 01:03:42 on dist-test-slave-5c7l"
server_key: "309050ae8e8029dcf684b1fd094ce18f"
server_key_iv: "4e9810b82daea2e63a2bdaacec78663d"
server_key_version: "encryptionkey@0"
I20260924 01:03:42.110853 19455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 01:03:42.115021 19469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:42.116931 19455 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 01:03:42.117110 19455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "1249c0a4743c417ab23ee48a021774e5"
format_stamp: "Formatted at 2026-09-24 01:03:42 on dist-test-slave-5c7l"
server_key: "309050ae8e8029dcf684b1fd094ce18f"
server_key_iv: "4e9810b82daea2e63a2bdaacec78663d"
server_key_version: "encryptionkey@0"
I20260924 01:03:42.117370 19455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:42.125675 19455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:42.126792 19455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:42.127161 19455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:42.128628 19455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:42.130810 19455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:42.130937 19455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:42.131062 19455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:42.131146 19455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:42.173619 19455 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:33715
I20260924 01:03:42.173669 19581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:33715 every 8 connection(s)
I20260924 01:03:42.175560 19455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:42.176631 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19455
I20260924 01:03:42.176800 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:03:42.177351 9209 external_mini_cluster.cc:1428] Setting key 1aba7a84a4aa03f6dcae9bd72366cba5
I20260924 01:03:42.186378 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:42.178623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:57059 (local address 127.8.254.126:36427)
0924 01:03:42.178829 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:42.178835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:42.180451 (+ 1616us) server_negotiation.cc:407] Connection header received
0924 01:03:42.183402 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:42.183408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:42.183489 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:42.183613 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:42.185456 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:42.185466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:42.185470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:42.185489 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:42.185547 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:42.185565 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:42.185683 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:42.185946 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 01:03:42.186008 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 01:03:42.187690 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:42.179246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:36427 (local address 127.8.254.67:57059)
0924 01:03:42.180303 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:42.180347 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:42.183042 (+ 2695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:42.183912 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:42.183925 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:42.184065 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:42.185066 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:03:42.185069 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:42.185123 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:03:42.185251 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:42.185686 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:42.185698 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:42.185707 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 01:03:42.186021 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 01:03:42.186336 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":337,"threads_started":1}
I20260924 01:03:42.189122 19582 heartbeater.cc:347] Connected to a master server at 127.8.254.126:36427
I20260924 01:03:42.189445 19582 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:42.190033 19582 heartbeater.cc:511] Master 127.8.254.126:36427 requested a full tablet report, sending...
I20260924 01:03:42.191702 19135 ts_manager.cc:194] Registered new tserver with Master: 1249c0a4743c417ab23ee48a021774e5 (127.8.254.67:33715)
I20260924 01:03:42.193158 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 01:03:42.199754 19323 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:03:42.198036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.8.254.126:36427)
0924 01:03:42.198273 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:42.198281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:42.198780 (+ 499us) server_negotiation.cc:407] Connection header received
0924 01:03:42.198998 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:42.199004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:42.199066 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:42.199184 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:42.199545 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41118: BlockingRecv error: recv got EOF from 127.0.0.1:41118 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:03:42.200822 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19194
I20260924 01:03:42.204305 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:42.201786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.8.254.65:39423)
0924 01:03:42.202358 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:42.202372 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:42.202398 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:03:42.202455 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:42.202461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:42.202666 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:42.202793 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:42.203712 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:42.203718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:42.203729 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:42.203745 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:42.203782 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:42.203795 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:42.203904 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:42.203994 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 01:03:42.204071 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
I20260924 01:03:42.205479 19316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:42.286866 19321 heartbeater.cc:503] Master 127.8.254.126:36427 was elected leader, sending a full tablet report...
I20260924 01:03:42.289378 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19194
I20260924 01:03:42.306922 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19325
I20260924 01:03:42.310756 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:42.308053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.8.254.66:35331)
0924 01:03:42.308619 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:42.308626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:42.308653 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:03:42.308716 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:42.308723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:42.308943 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:42.309075 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:42.309977 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:42.309982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:42.309993 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:42.310010 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:42.310050 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:42.310064 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:42.310179 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:42.310335 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 01:03:42.310454 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
I20260924 01:03:42.312084 19447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:42.393620 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19325
I20260924 01:03:42.409686 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19455
I20260924 01:03:42.413774 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:42.410730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.8.254.67:33715)
0924 01:03:42.411307 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:42.411314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:42.411345 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:03:42.411413 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:42.411419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:42.411658 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:42.411838 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:42.412988 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:42.412994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:42.413006 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:03:42.413017 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:42.413054 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:03:42.413069 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:42.413270 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:42.413349 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 01:03:42.413469 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":287,"threads_started":1}
I20260924 01:03:42.415278 19577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:42.496541 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19455
I20260924 01:03:42.512578 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19104
I20260924 01:03:42.513790 19165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:42.616802 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19104
2026-09-24T01:03:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2415 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:03:42 dist-test-slave-5c7l krb5kdc[19600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:03:42 dist-test-slave-5c7l krb5kdc[19600](info): set up 2 sockets
Sep 24 01:03:42 dist-test-slave-5c7l krb5kdc[19600](info): commencing operation
krb5kdc: starting...
W20260924 01:03:43.752321 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:03:43 dist-test-slave-5c7l krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:03:43.994674 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45389
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:44.329718 19613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:44.330039 19613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:44.330133 19613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:44.337683 19613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:03:44.337807 19613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:44.337855 19613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:44.337913 19613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:03:44.337953 19613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:03:44.348321 19613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:44.349704 19613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:44.351040 19613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:44.363641 19621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:44.363755 19619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:44.363739 19618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:44.364526 19613 server_base.cc:1034] running on GCE node
I20260924 01:03:44.365432 19613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:44.367128 19613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:44.368407 19613 hybrid_clock.cc:648] HybridClock initialized: now 1790211824368359 us; error 37 us; skew 500 ppm
I20260924 01:03:44.368810 19613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:44 dist-test-slave-5c7l krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:44.377079 19613 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:03:44.378772 19613 webserver.cc:460] Webserver started at http://127.8.254.126:35687/ using document root <none> and password file <none>
I20260924 01:03:44.379441 19613 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:44.379544 19613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:44.379797 19613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:44.382831 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "febe8ba5567f49778d6f78572dd86881"
format_stamp: "Formatted at 2026-09-24 01:03:44 on dist-test-slave-5c7l"
server_key: "c19700ae8f804de38379eafdb8446607"
server_key_iv: "7f0735f00c1ea17ffc620a856e8245a9"
server_key_version: "encryptionkey@0"
I20260924 01:03:44.383592 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "febe8ba5567f49778d6f78572dd86881"
format_stamp: "Formatted at 2026-09-24 01:03:44 on dist-test-slave-5c7l"
server_key: "c19700ae8f804de38379eafdb8446607"
server_key_iv: "7f0735f00c1ea17ffc620a856e8245a9"
server_key_version: "encryptionkey@0"
I20260924 01:03:44.388741 19613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:03:44.392095 19628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:44.393435 19613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:03:44.393576 19613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "febe8ba5567f49778d6f78572dd86881"
format_stamp: "Formatted at 2026-09-24 01:03:44 on dist-test-slave-5c7l"
server_key: "c19700ae8f804de38379eafdb8446607"
server_key_iv: "7f0735f00c1ea17ffc620a856e8245a9"
server_key_version: "encryptionkey@0"
I20260924 01:03:44.393801 19613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:44.409561 19613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:44.413544 19613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:44.413779 19613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:44.432595 19613 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45389
I20260924 01:03:44.432632 19679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45389 every 8 connection(s)
I20260924 01:03:44.434064 19613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:03:44.437764 19680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:44.438493 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19613
I20260924 01:03:44.438716 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:03:44.439363 9209 external_mini_cluster.cc:1428] Setting key ebbd2a84a5aa67c9a953c0d7926e4c2d
I20260924 01:03:44.452021 19680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881: Bootstrap starting.
Sep 24 01:03:44 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:44.455919 19680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:44.457007 19680 log.cc:824] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:44.460222 19680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881: No bootstrap required, opened a new log
I20260924 01:03:44.463423 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:44.441532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.8.254.126:45389)
0924 01:03:44.442392 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:44.442403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:44.442444 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:03:44.444062 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:44.444083 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:44.444479 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:44.444797 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:44.446621 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:44.447888 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:44.449709 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:44.450561 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:44.450690 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:44.455485 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:44.455501 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:44.455517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:44.455562 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:44.459189 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:44.459920 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:44.459933 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:44.459939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:44.460110 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:44.460582 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:44.460588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:44.460591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:44.461138 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:44.461466 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:44.461939 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 01:03:44.462410 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":402,"threads_started":1}
I20260924 01:03:44.468333 19680 raft_consensus.cc:348] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febe8ba5567f49778d6f78572dd86881" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45389 } }
I20260924 01:03:44.468712 19680 raft_consensus.cc:374] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:44.468811 19680 raft_consensus.cc:729] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febe8ba5567f49778d6f78572dd86881, State: Initialized, Role: FOLLOWER
I20260924 01:03:44.469396 19680 consensus_queue.cc:260] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febe8ba5567f49778d6f78572dd86881" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45389 } }
I20260924 01:03:44.469646 19680 raft_consensus.cc:388] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:03:44.469798 19680 raft_consensus.cc:482] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:03:44.469983 19680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:44.472218 19680 raft_consensus.cc:504] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febe8ba5567f49778d6f78572dd86881" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45389 } }
I20260924 01:03:44.472726 19680 leader_election.cc:302] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: febe8ba5567f49778d6f78572dd86881; no voters:
I20260924 01:03:44.473353 19680 leader_election.cc:288] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:03:44.473578 19685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:44.474599 19685 raft_consensus.cc:686] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [term 1 LEADER]: Becoming Leader. State: Replica: febe8ba5567f49778d6f78572dd86881, State: Running, Role: LEADER
I20260924 01:03:44.475117 19685 consensus_queue.cc:237] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febe8ba5567f49778d6f78572dd86881" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45389 } }
I20260924 01:03:44.475622 19680 sys_catalog.cc:564] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:03:44.478106 19686 sys_catalog.cc:455] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "febe8ba5567f49778d6f78572dd86881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febe8ba5567f49778d6f78572dd86881" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45389 } } }
I20260924 01:03:44.478466 19686 sys_catalog.cc:458] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:44.478071 19687 sys_catalog.cc:455] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader febe8ba5567f49778d6f78572dd86881. Latest consensus state: current_term: 1 leader_uuid: "febe8ba5567f49778d6f78572dd86881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febe8ba5567f49778d6f78572dd86881" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45389 } } }
I20260924 01:03:44.479496 19687 sys_catalog.cc:458] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881 [sys.catalog]: This master's current role is: LEADER
I20260924 01:03:44.481179 19690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:03:44.486970 19690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:03:44.495767 19690 catalog_manager.cc:1347] Generated new cluster ID: abe9693bffb1480988118d3f1979cfed
I20260924 01:03:44.495903 19690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:03:44.519102 19690 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:03:44.520468 19690 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:03:44.529894 19690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881: Generated new TSK 0
I20260924 01:03:44.530870 19690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:03:44.594779 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45389
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:44.934235 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:44.934585 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:44.934769 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:44.942092 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:44.942286 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:44.942480 19706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:03:44.952299 19706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:44.953719 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:44.955164 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:44.969564 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:44.969733 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:44.969719 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:44.970532 19706 server_base.cc:1034] running on GCE node
I20260924 01:03:44.971180 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:44.972199 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:44.973475 19706 hybrid_clock.cc:648] HybridClock initialized: now 1790211824973411 us; error 37 us; skew 500 ppm
I20260924 01:03:44.973822 19706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:44 dist-test-slave-5c7l krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:44.977242 19706 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:03:44.979135 19706 webserver.cc:460] Webserver started at http://127.8.254.65:35983/ using document root <none> and password file <none>
I20260924 01:03:44.980021 19706 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:44.980190 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:44.980561 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:44.984102 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "db9c7b766d9d4946a2e7463068cc8abc"
format_stamp: "Formatted at 2026-09-24 01:03:44 on dist-test-slave-5c7l"
server_key: "98d921e5fa78b385892f0fce93088c93"
server_key_iv: "ec52a1e799036847866a6bcb45b2c95c"
server_key_version: "encryptionkey@0"
I20260924 01:03:44.984992 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "db9c7b766d9d4946a2e7463068cc8abc"
format_stamp: "Formatted at 2026-09-24 01:03:44 on dist-test-slave-5c7l"
server_key: "98d921e5fa78b385892f0fce93088c93"
server_key_iv: "ec52a1e799036847866a6bcb45b2c95c"
server_key_version: "encryptionkey@0"
I20260924 01:03:44.990674 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 01:03:44.994539 19721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:44.996089 19706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:03:44.996281 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "db9c7b766d9d4946a2e7463068cc8abc"
format_stamp: "Formatted at 2026-09-24 01:03:44 on dist-test-slave-5c7l"
server_key: "98d921e5fa78b385892f0fce93088c93"
server_key_iv: "ec52a1e799036847866a6bcb45b2c95c"
server_key_version: "encryptionkey@0"
I20260924 01:03:44.996569 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:45.017503 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:45.022414 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:45.022748 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:45.023936 19706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:45.025436 19706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:45.025559 19706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:45.025733 19706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:45.025817 19706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:45.062254 19706 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:37963
I20260924 01:03:45.062356 19833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:37963 every 8 connection(s)
I20260924 01:03:45.063699 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:03:45.071105 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19706
I20260924 01:03:45.071331 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:03:45.071954 9209 external_mini_cluster.cc:1428] Setting key b2f30bcfd05299afa30525e4b922a6b9
Sep 24 01:03:45 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:45.084622 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:45.066502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:33219 (local address 127.8.254.126:45389)
0924 01:03:45.066971 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:45.066976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:45.067945 (+ 969us) server_negotiation.cc:407] Connection header received
0924 01:03:45.069776 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:45.069782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:45.069900 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:45.070027 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:45.071715 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.072487 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.074398 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.074796 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.074945 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:45.078700 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:45.078705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:45.078709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:45.078741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:45.081341 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:45.082147 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:45.082153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:45.082157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:45.082372 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:45.082889 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:45.082894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:45.082898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:45.083161 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:45.083322 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:45.084004 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 01:03:45.084234 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260924 01:03:45.085609 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:45.067061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45389 (local address 127.8.254.65:33219)
0924 01:03:45.067782 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:45.067823 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:45.069499 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:45.070339 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:45.070348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:45.070860 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:45.071545 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:45.071565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.072653 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.072670 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:45.074174 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:45.074187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.075043 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.075051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:45.075301 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:45.076000 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:45.076026 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:45.078466 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:45.081499 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:45.081507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:45.081525 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:45.082019 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:45.082550 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:45.082556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:45.082560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:45.082745 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:45.083297 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:45.083305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:45.083640 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 01:03:45.083917 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 01:03:45.084445 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":284,"threads_started":1}
I20260924 01:03:45.087255 19834 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45389
I20260924 01:03:45.087634 19834 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:45.088387 19834 heartbeater.cc:511] Master 127.8.254.126:45389 requested a full tablet report, sending...
I20260924 01:03:45.090854 19645 ts_manager.cc:194] Registered new tserver with Master: db9c7b766d9d4946a2e7463068cc8abc (127.8.254.65:37963)
I20260924 01:03:45.093173 19645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:33219
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:03:45.125926 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45389
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:45.460448 19840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:45.460744 19840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:45.460899 19840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:45.468220 19840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:45.468330 19840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:45.468540 19840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:03:45.478530 19840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:45.479898 19840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:45.481730 19840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:45.498333 19846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:45.498494 19845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:45.498490 19848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:45.499490 19840 server_base.cc:1034] running on GCE node
I20260924 01:03:45.500145 19840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:45.501240 19840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:45.502516 19840 hybrid_clock.cc:648] HybridClock initialized: now 1790211825502468 us; error 55 us; skew 500 ppm
I20260924 01:03:45.502862 19840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:45 dist-test-slave-5c7l krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:45.506019 19840 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:03:45.507550 19840 webserver.cc:460] Webserver started at http://127.8.254.66:37141/ using document root <none> and password file <none>
I20260924 01:03:45.508247 19840 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:45.508400 19840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:45.508702 19840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:45.511659 19840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "c9c1e1300036448eb8fac864d61a0cc7"
format_stamp: "Formatted at 2026-09-24 01:03:45 on dist-test-slave-5c7l"
server_key: "ed14f4d1903c885a795c5fe1ba9d6264"
server_key_iv: "c2a8c4cea5efc466b59580203a898e2b"
server_key_version: "encryptionkey@0"
I20260924 01:03:45.512480 19840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "c9c1e1300036448eb8fac864d61a0cc7"
format_stamp: "Formatted at 2026-09-24 01:03:45 on dist-test-slave-5c7l"
server_key: "ed14f4d1903c885a795c5fe1ba9d6264"
server_key_iv: "c2a8c4cea5efc466b59580203a898e2b"
server_key_version: "encryptionkey@0"
I20260924 01:03:45.517606 19840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 01:03:45.521219 19855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:45.522521 19840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:45.522701 19840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "c9c1e1300036448eb8fac864d61a0cc7"
format_stamp: "Formatted at 2026-09-24 01:03:45 on dist-test-slave-5c7l"
server_key: "ed14f4d1903c885a795c5fe1ba9d6264"
server_key_iv: "c2a8c4cea5efc466b59580203a898e2b"
server_key_version: "encryptionkey@0"
I20260924 01:03:45.522898 19840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:45.556825 19840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:45.561535 19840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:45.561887 19840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:45.562922 19840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:45.564437 19840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:45.564543 19840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:45.564702 19840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:45.564774 19840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:45.598444 19840 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:41329
I20260924 01:03:45.598537 19967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:41329 every 8 connection(s)
I20260924 01:03:45.600155 19840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:03:45.610154 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19840
I20260924 01:03:45.610402 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:03:45.610983 9209 external_mini_cluster.cc:1428] Setting key c73edefbba16a270537675cb90b7484e
Sep 24 01:03:45 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:45.621803 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:45.603063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:36529 (local address 127.8.254.126:45389)
0924 01:03:45.603573 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:45.603578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:45.604385 (+ 807us) server_negotiation.cc:407] Connection header received
0924 01:03:45.606243 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:45.606249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:45.606349 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:45.606503 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:45.608060 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.608782 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.610403 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.611051 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.611201 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:45.615623 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:45.615629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:45.615632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:45.615665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:45.618187 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:45.619264 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:45.619273 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:45.619280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:45.619436 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:45.619858 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:45.619865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:45.619871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:45.620246 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:45.620413 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:45.621094 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 01:03:45.621351 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":261,"threads_started":1}
I20260924 01:03:45.622637 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:45.603522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45389 (local address 127.8.254.66:36529)
0924 01:03:45.604222 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:45.604260 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:45.605946 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:45.606751 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:45.606779 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:45.607319 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:45.607876 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:45.607903 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.608961 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.608965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:45.610171 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:45.610184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:45.611275 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:45.611281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:45.611529 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:45.612311 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:45.612338 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:45.615264 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:45.618403 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:45.618414 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:45.618434 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:45.619097 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:45.619550 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:45.619553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:45.619556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:45.619742 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:45.620368 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:45.620374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:45.620707 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 01:03:45.620948 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 01:03:45.621349 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":280,"threads_started":1}
I20260924 01:03:45.624173 19968 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45389
I20260924 01:03:45.624500 19968 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:45.625265 19968 heartbeater.cc:511] Master 127.8.254.126:45389 requested a full tablet report, sending...
I20260924 01:03:45.626804 19645 ts_manager.cc:194] Registered new tserver with Master: c9c1e1300036448eb8fac864d61a0cc7 (127.8.254.66:41329)
I20260924 01:03:45.627748 19645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:36529
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:03:45.666019 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45389
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:03:46.002784 19974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:03:46.003083 19974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:03:46.003382 19974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:03:46.011030 19974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:03:46.011143 19974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:03:46.011353 19974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:03:46.021545 19974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:03:46.022873 19974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:03:46.024214 19974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:03:46.037887 19979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:46.037935 19980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:03:46.037865 19982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:03:46.038882 19974 server_base.cc:1034] running on GCE node
I20260924 01:03:46.039518 19974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:03:46.040489 19974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:03:46.041728 19974 hybrid_clock.cc:648] HybridClock initialized: now 1790211826041678 us; error 42 us; skew 500 ppm
I20260924 01:03:46.042063 19974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:03:46 dist-test-slave-5c7l krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:03:46.045306 19974 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:03:46.046850 19974 webserver.cc:460] Webserver started at http://127.8.254.67:40847/ using document root <none> and password file <none>
I20260924 01:03:46.047565 19974 fs_manager.cc:359] Metadata directory not provided
I20260924 01:03:46.047716 19974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:03:46.048029 19974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:03:46.051126 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "76c49db7be914354bcea2b489b1588ba"
format_stamp: "Formatted at 2026-09-24 01:03:46 on dist-test-slave-5c7l"
server_key: "8718fb5d2629d0f19242c013e8d11194"
server_key_iv: "9a1b11e8f695c72047f9434e0ff0913d"
server_key_version: "encryptionkey@0"
I20260924 01:03:46.051926 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "76c49db7be914354bcea2b489b1588ba"
format_stamp: "Formatted at 2026-09-24 01:03:46 on dist-test-slave-5c7l"
server_key: "8718fb5d2629d0f19242c013e8d11194"
server_key_iv: "9a1b11e8f695c72047f9434e0ff0913d"
server_key_version: "encryptionkey@0"
I20260924 01:03:46.057469 19974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:03:46.061105 19989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:46.062852 19974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:03:46.063030 19974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "76c49db7be914354bcea2b489b1588ba"
format_stamp: "Formatted at 2026-09-24 01:03:46 on dist-test-slave-5c7l"
server_key: "8718fb5d2629d0f19242c013e8d11194"
server_key_iv: "9a1b11e8f695c72047f9434e0ff0913d"
server_key_version: "encryptionkey@0"
I20260924 01:03:46.063269 19974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:03:46.083065 19974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:03:46.086905 19974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:03:46.087172 19974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:03:46.088058 19974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:03:46.089412 19974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:03:46.089488 19974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:46.089588 19974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:03:46.089632 19974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:03:46.096513 19834 heartbeater.cc:503] Master 127.8.254.126:45389 was elected leader, sending a full tablet report...
I20260924 01:03:46.122905 19974 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:35751
I20260924 01:03:46.122956 20101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:35751 every 8 connection(s)
I20260924 01:03:46.124219 19974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:03:46.130175 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 19974
I20260924 01:03:46.130384 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:03:46.130887 9209 external_mini_cluster.cc:1428] Setting key ad32d1770c03fadbb868ea39c2fb3bbe
Sep 24 01:03:46 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:03:46.144507 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.126931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:40953 (local address 127.8.254.126:45389)
0924 01:03:46.127383 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.127390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.128441 (+ 1051us) server_negotiation.cc:407] Connection header received
0924 01:03:46.130227 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.130234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.130340 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.130469 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:46.132143 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.133248 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.135013 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.135400 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.135541 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.138948 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:46.138953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:46.138956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:46.138988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:46.141388 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.142092 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.142098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.142102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.142290 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.142719 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.142725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.142728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.143015 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:46.143184 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.143879 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.144116 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":204,"threads_started":1}
I20260924 01:03:46.145655 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.127545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45389 (local address 127.8.254.67:40953)
0924 01:03:46.128272 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.128319 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:46.129928 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:46.130721 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:46.130730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:46.131285 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:03:46.131982 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.132001 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.133415 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.133420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.134855 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.134868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.135640 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.135648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.135901 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.136488 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:03:46.136549 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:03:46.138726 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:03:46.141538 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:46.141546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:46.141564 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:46.141955 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:46.142400 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:03:46.142405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:03:46.142408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:03:46.142602 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:03:46.143153 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:03:46.143161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:03:46.143507 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 01:03:46.143799 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 01:03:46.144335 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":255,"threads_started":1}
I20260924 01:03:46.147012 20102 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45389
I20260924 01:03:46.147294 20102 heartbeater.cc:464] Registering TS with master...
I20260924 01:03:46.148010 20102 heartbeater.cc:511] Master 127.8.254.126:45389 requested a full tablet report, sending...
I20260924 01:03:46.149402 19645 ts_manager.cc:194] Registered new tserver with Master: 76c49db7be914354bcea2b489b1588ba (127.8.254.67:35751)
I20260924 01:03:46.150290 19645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:40953
I20260924 01:03:46.159269 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:03:46.176926 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.164171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.8.254.126:45389)
0924 01:03:46.164401 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.164409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.164910 (+ 501us) server_negotiation.cc:407] Connection header received
0924 01:03:46.165177 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.165181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.165250 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.165366 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:46.166942 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.167677 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.169010 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.169429 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.169572 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.171820 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:46.171825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:46.171829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:46.171862 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:46.174249 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.174892 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.174906 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.174910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.175044 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.175415 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.175422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.175427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.175769 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:46.176070 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.176386 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.176607 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 01:03:46.181005 19645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:03:46.187923 19645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:03:46.210425 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.201736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.8.254.65:37963)
0924 01:03:46.202788 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.202796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.202823 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:03:46.202913 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.202920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.203156 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.204039 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:46.204234 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.205184 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.208217 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.209209 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.209373 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.209609 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.209874 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.210041 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":740,"threads_started":1}
I20260924 01:03:46.210536 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.201370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60506)
0924 01:03:46.202044 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.202068 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:46.202229 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:46.203370 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:46.203374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:46.203397 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:46.203850 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.203861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.205319 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.205327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.208034 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.208056 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.209416 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.209422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.209607 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.209675 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 01:03:46.209819 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 01:03:46.210024 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":26,"thread_start_us":373,"threads_started":1}
I20260924 01:03:46.211004 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.199987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41329 (local address 127.0.0.1:35060)
0924 01:03:46.200914 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.200949 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:46.201116 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:46.202446 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:46.202456 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:46.202494 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:46.202881 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.202899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.205484 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.205493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.208158 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.208172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.209983 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.209991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.210234 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.210269 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:03:46.210344 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 01:03:46.210568 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":81,"thread_start_us":571,"threads_started":1}
I20260924 01:03:46.212899 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.200296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.8.254.66:41329)
0924 01:03:46.201766 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.201775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.201804 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:03:46.201907 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.201917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.202176 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.203115 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:46.203363 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.205301 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.208317 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.209757 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.211284 (+ 1527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.211495 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.211573 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.211728 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":1127,"threads_started":1}
I20260924 01:03:46.214715 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.202067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:35751 (local address 127.0.0.1:50558)
0924 01:03:46.206423 (+ 4356us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.206440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:46.206567 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:46.207374 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:46.207378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:46.207403 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:46.207909 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.207920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.209888 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.209894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.212456 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.212473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.213876 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.213883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.214050 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.214092 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 01:03:46.214220 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 01:03:46.214347 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4227,"mutex_wait_us":1,"thread_start_us":4047,"threads_started":1}
I20260924 01:03:46.214752 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.202639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.8.254.67:35751)
0924 01:03:46.205825 (+ 3186us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.205833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.206550 (+ 717us) server_negotiation.cc:407] Connection header received
0924 01:03:46.206709 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.206716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.206939 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.207127 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:46.208118 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.209713 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.212631 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.213670 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.213816 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.214040 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.214257 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.214414 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3011,"thread_start_us":2834,"threads_started":1}
I20260924 01:03:46.215024 19769 tablet_service.cc:1467] Processing CreateTablet for tablet 94296bf912d74fd798ba40e7fcbaf688 (DEFAULT_TABLE table=test-table [id=f6df762c55a84f8082c677a9f03136fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:46.216658 19769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94296bf912d74fd798ba40e7fcbaf688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:46.216374 19903 tablet_service.cc:1467] Processing CreateTablet for tablet 94296bf912d74fd798ba40e7fcbaf688 (DEFAULT_TABLE table=test-table [id=f6df762c55a84f8082c677a9f03136fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:46.217942 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94296bf912d74fd798ba40e7fcbaf688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:46.217763 20037 tablet_service.cc:1467] Processing CreateTablet for tablet 94296bf912d74fd798ba40e7fcbaf688 (DEFAULT_TABLE table=test-table [id=f6df762c55a84f8082c677a9f03136fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:03:46.219112 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94296bf912d74fd798ba40e7fcbaf688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:03:46.230962 20121 tablet_bootstrap.cc:492] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Bootstrap starting.
I20260924 01:03:46.234956 20122 tablet_bootstrap.cc:492] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Bootstrap starting.
I20260924 01:03:46.235807 20121 tablet_bootstrap.cc:654] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:46.237283 20121 log.cc:824] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:46.238772 20122 tablet_bootstrap.cc:654] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:46.238807 20123 tablet_bootstrap.cc:492] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Bootstrap starting.
I20260924 01:03:46.240187 20122 log.cc:824] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:46.240729 20121 tablet_bootstrap.cc:492] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: No bootstrap required, opened a new log
I20260924 01:03:46.241073 20121 ts_tablet_manager.cc:1397] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 01:03:46.242702 20123 tablet_bootstrap.cc:654] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Neither blocks nor log segments found. Creating new log.
I20260924 01:03:46.243973 20123 log.cc:824] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Log is configured to *not* fsync() on all Append() calls
I20260924 01:03:46.244890 20122 tablet_bootstrap.cc:492] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: No bootstrap required, opened a new log
I20260924 01:03:46.245163 20122 ts_tablet_manager.cc:1397] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 01:03:46.247157 20123 tablet_bootstrap.cc:492] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: No bootstrap required, opened a new log
I20260924 01:03:46.247417 20123 ts_tablet_manager.cc:1397] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 01:03:46.250258 20121 raft_consensus.cc:348] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.250685 20121 raft_consensus.cc:374] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:46.250825 20121 raft_consensus.cc:729] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c49db7be914354bcea2b489b1588ba, State: Initialized, Role: FOLLOWER
I20260924 01:03:46.251575 20121 consensus_queue.cc:260] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.254127 20102 heartbeater.cc:503] Master 127.8.254.126:45389 was elected leader, sending a full tablet report...
I20260924 01:03:46.254927 20121 ts_tablet_manager.cc:1428] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 01:03:46.256031 20122 raft_consensus.cc:348] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.256506 20122 raft_consensus.cc:374] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:46.256608 20122 raft_consensus.cc:729] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9c7b766d9d4946a2e7463068cc8abc, State: Initialized, Role: FOLLOWER
I20260924 01:03:46.257283 20122 consensus_queue.cc:260] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.258016 20123 raft_consensus.cc:348] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.258527 20123 raft_consensus.cc:374] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:03:46.258639 20123 raft_consensus.cc:729] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c1e1300036448eb8fac864d61a0cc7, State: Initialized, Role: FOLLOWER
I20260924 01:03:46.259174 20122 ts_tablet_manager.cc:1428] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 01:03:46.259255 20123 consensus_queue.cc:260] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.260604 19968 heartbeater.cc:503] Master 127.8.254.126:45389 was elected leader, sending a full tablet report...
I20260924 01:03:46.260949 20123 ts_tablet_manager.cc:1428] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 01:03:46.301064 20127 raft_consensus.cc:482] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:03:46.301424 20127 raft_consensus.cc:504] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.303400 20127 leader_election.cc:288] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9c1e1300036448eb8fac864d61a0cc7 (127.8.254.66:41329), db9c7b766d9d4946a2e7463068cc8abc (127.8.254.65:37963)
I20260924 01:03:46.308936 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.303798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:52089 (local address 127.8.254.66:41329)
0924 01:03:46.304032 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.304038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.304057 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:03:46.304145 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.304151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.304244 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.304361 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:46.304883 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.305612 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.307245 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.308042 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.308216 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.308289 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.308548 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.308668 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:03:46.308926 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.303659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41329 (local address 127.8.254.67:52089)
0924 01:03:46.303874 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.303895 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:46.304025 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:46.304551 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:46.304554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:46.304576 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:46.304736 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.304744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.305749 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.305756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.307085 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.307097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.308191 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.308200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.308383 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.308407 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 01:03:46.308520 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 01:03:46.308653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":21}
I20260924 01:03:46.309908 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.304049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.8.254.67:59765)
0924 01:03:46.304658 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.304675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:03:46.304817 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:03:46.305148 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:03:46.305152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:03:46.305177 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:03:46.305480 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.305488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.306567 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.306572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.307959 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:03:46.307970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.309266 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.309272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:03:46.309445 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.309477 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:03:46.309568 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 01:03:46.309657 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":19,"thread_start_us":297,"threads_started":1}
I20260924 01:03:46.310007 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.304269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:59765 (local address 127.8.254.65:37963)
0924 01:03:46.304478 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.304483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.304759 (+ 276us) server_negotiation.cc:407] Connection header received
0924 01:03:46.304955 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.304960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.305030 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.305125 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:03:46.305633 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.306427 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.308114 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.308958 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.309151 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.309213 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.309636 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.309772 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 01:03:46.310060 19923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94296bf912d74fd798ba40e7fcbaf688" candidate_uuid: "76c49db7be914354bcea2b489b1588ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c1e1300036448eb8fac864d61a0cc7" is_pre_election: true
I20260924 01:03:46.310702 19923 raft_consensus.cc:2454] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c49db7be914354bcea2b489b1588ba in term 0.
I20260924 01:03:46.311098 19789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94296bf912d74fd798ba40e7fcbaf688" candidate_uuid: "76c49db7be914354bcea2b489b1588ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9c7b766d9d4946a2e7463068cc8abc" is_pre_election: true
I20260924 01:03:46.311368 19990 leader_election.cc:302] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c49db7be914354bcea2b489b1588ba, c9c1e1300036448eb8fac864d61a0cc7; no voters:
I20260924 01:03:46.311666 19789 raft_consensus.cc:2454] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c49db7be914354bcea2b489b1588ba in term 0.
I20260924 01:03:46.311822 20127 raft_consensus.cc:2781] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:03:46.311988 20127 raft_consensus.cc:482] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:03:46.312114 20127 raft_consensus.cc:3037] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:46.314465 20127 raft_consensus.cc:504] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.315073 20127 leader_election.cc:288] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [CANDIDATE]: Term 1 election: Requested vote from peers c9c1e1300036448eb8fac864d61a0cc7 (127.8.254.66:41329), db9c7b766d9d4946a2e7463068cc8abc (127.8.254.65:37963)
I20260924 01:03:46.315451 19923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94296bf912d74fd798ba40e7fcbaf688" candidate_uuid: "76c49db7be914354bcea2b489b1588ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c1e1300036448eb8fac864d61a0cc7"
I20260924 01:03:46.315532 19789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94296bf912d74fd798ba40e7fcbaf688" candidate_uuid: "76c49db7be914354bcea2b489b1588ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9c7b766d9d4946a2e7463068cc8abc"
I20260924 01:03:46.315657 19923 raft_consensus.cc:3037] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:03:46.315765 19789 raft_consensus.cc:3037] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 0 FOLLOWER]: Advancing to term 1
W20260924 01:03:46.316278 19835 tablet.cc:2369] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:46.318154 19789 raft_consensus.cc:2454] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c49db7be914354bcea2b489b1588ba in term 1.
I20260924 01:03:46.318444 19923 raft_consensus.cc:2454] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c49db7be914354bcea2b489b1588ba in term 1.
I20260924 01:03:46.318620 19990 leader_election.cc:302] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c49db7be914354bcea2b489b1588ba, db9c7b766d9d4946a2e7463068cc8abc; no voters:
I20260924 01:03:46.318930 20127 raft_consensus.cc:2781] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:03:46.319381 20127 raft_consensus.cc:686] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 1 LEADER]: Becoming Leader. State: Replica: 76c49db7be914354bcea2b489b1588ba, State: Running, Role: LEADER
I20260924 01:03:46.319857 20127 consensus_queue.cc:237] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } }
I20260924 01:03:46.324891 19643 catalog_manager.cc:5495] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c49db7be914354bcea2b489b1588ba (127.8.254.67). New cstate: current_term: 1 leader_uuid: "76c49db7be914354bcea2b489b1588ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c49db7be914354bcea2b489b1588ba" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 35751 } health_report { overall_health: HEALTHY } } }
W20260924 01:03:46.352571 19969 tablet.cc:2369] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:46.368952 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.363687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.8.254.67:35751)
0924 01:03:46.363872 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.363877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.363894 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:03:46.364042 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.364047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.364129 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.364261 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:03:46.364602 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.365332 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.366761 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.367163 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.367259 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.367751 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:03:46.368324 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:03:46.368442 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.368556 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.368639 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 01:03:46.375753 20103 tablet.cc:2369] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:03:46.378162 19789 raft_consensus.cc:1261] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 1 FOLLOWER]: Refusing update from remote peer 76c49db7be914354bcea2b489b1588ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:46.378242 19923 raft_consensus.cc:1261] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 1 FOLLOWER]: Refusing update from remote peer 76c49db7be914354bcea2b489b1588ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:03:46.379243 20127 consensus_queue.cc:1035] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9c1e1300036448eb8fac864d61a0cc7" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:46.379565 20131 consensus_queue.cc:1035] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "db9c7b766d9d4946a2e7463068cc8abc" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:03:46.390908 20134 mvcc.cc:205] Tried to move back new op lower bound from 7332707640831184896 to 7332707640610926592. Current Snapshot: MvccSnapshot[applied={T|T < 7332707640831184896}]
I20260924 01:03:46.394093 20136 mvcc.cc:205] Tried to move back new op lower bound from 7332707640831184896 to 7332707640610926592. Current Snapshot: MvccSnapshot[applied={T|T < 7332707640831184896}]
I20260924 01:03:46.405200 19643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:46.405478 19643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:03:46.409248 19643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P febe8ba5567f49778d6f78572dd86881: Sending DeleteTablet for 3 replicas of tablet 94296bf912d74fd798ba40e7fcbaf688
I20260924 01:03:46.410478 19903 tablet_service.cc:1514] Processing DeleteTablet for tablet 94296bf912d74fd798ba40e7fcbaf688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:46 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:35060
I20260924 01:03:46.410524 20037 tablet_service.cc:1514] Processing DeleteTablet for tablet 94296bf912d74fd798ba40e7fcbaf688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:46 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:50558
I20260924 01:03:46.411544 19769 tablet_service.cc:1514] Processing DeleteTablet for tablet 94296bf912d74fd798ba40e7fcbaf688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:03:46 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:60506
I20260924 01:03:46.411679 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19706
I20260924 01:03:46.422889 20144 tablet_replica.cc:330] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: stopping tablet replica
I20260924 01:03:46.424233 20144 raft_consensus.cc:2229] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:03:46.425701 20144 raft_consensus.cc:2258] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:46.428653 20145 tablet_replica.cc:330] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: stopping tablet replica
I20260924 01:03:46.429234 20145 raft_consensus.cc:2229] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:46.429546 20144 ts_tablet_manager.cc:1905] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:46.429870 20145 raft_consensus.cc:2258] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 01:03:46 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:03:46.437875 20144 ts_tablet_manager.cc:1918] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:46.438166 20144 log.cc:1196] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/94296bf912d74fd798ba40e7fcbaf688
I20260924 01:03:46.439903 20144 ts_tablet_manager.cc:1939] T 94296bf912d74fd798ba40e7fcbaf688 P 76c49db7be914354bcea2b489b1588ba: Deleting consensus metadata
I20260924 01:03:46.441751 19629 catalog_manager.cc:4841] TS 76c49db7be914354bcea2b489b1588ba (127.8.254.67:35751): tablet 94296bf912d74fd798ba40e7fcbaf688 (table test-table [id=f6df762c55a84f8082c677a9f03136fd]) successfully deleted
I20260924 01:03:46.442557 20145 ts_tablet_manager.cc:1905] T 94296bf912d74fd798ba40e7fcbaf688 P db9c7b766d9d4946a2e7463068cc8abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:46.445855 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.413263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.8.254.65:37963)
0924 01:03:46.413454 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.413461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.425518 (+ 12057us) server_negotiation.cc:407] Connection header received
0924 01:03:46.425723 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.425729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.425851 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.425968 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:46.428031 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.429424 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.431350 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.432009 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.432137 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.437862 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:46.437885 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:46.437907 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:46.437961 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:46.441958 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.442916 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.442926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.442930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.443147 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.443651 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.443656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.443660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.444180 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:46.444513 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.445177 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.445436 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:03:46.447690 19829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:46.448300 20143 tablet_replica.cc:330] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: stopping tablet replica
I20260924 01:03:46.448722 20143 raft_consensus.cc:2229] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:03:46.449223 20143 raft_consensus.cc:2258] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:03:46.451581 20143 ts_tablet_manager.cc:1905] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:03:46.458676 20143 ts_tablet_manager.cc:1918] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:03:46.458931 20143 log.cc:1196] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/94296bf912d74fd798ba40e7fcbaf688
I20260924 01:03:46.459545 20143 ts_tablet_manager.cc:1939] T 94296bf912d74fd798ba40e7fcbaf688 P c9c1e1300036448eb8fac864d61a0cc7: Deleting consensus metadata
I20260924 01:03:46.461592 19629 catalog_manager.cc:4841] TS c9c1e1300036448eb8fac864d61a0cc7 (127.8.254.66:41329): tablet 94296bf912d74fd798ba40e7fcbaf688 (table test-table [id=f6df762c55a84f8082c677a9f03136fd]) successfully deleted
I20260924 01:03:46.547447 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19706
W20260924 01:03:46.565393 19629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.254.65:37963 (error 108)
I20260924 01:03:46.565609 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19840
I20260924 01:03:46.567354 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.566848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60518)
0924 01:03:46.567039 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.567174 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 01:03:46.567693 19629 catalog_manager.cc:4584] TS db9c7b766d9d4946a2e7463068cc8abc (127.8.254.65:37963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94296bf912d74fd798ba40e7fcbaf688: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Sep 24 01:03:46 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:03:46.581669 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.566432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.8.254.66:41329)
0924 01:03:46.566617 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.566624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.566995 (+ 371us) server_negotiation.cc:407] Connection header received
0924 01:03:46.567076 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.567080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.567144 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.567271 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:46.568724 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.569436 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.570883 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.571276 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.571411 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.575703 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:46.575718 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:46.575735 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:46.575769 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:46.578799 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.579515 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.579523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.579525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.579651 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.580040 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.580052 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.580055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.580435 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:46.580746 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.581082 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.581307 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:03:46.582396 19963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:46.621747 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.621176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60524)
0924 01:03:46.621429 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.621545 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 01:03:46.622963 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.622548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60540)
0924 01:03:46.622722 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.622823 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 01:03:46.676869 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19840
I20260924 01:03:46.683637 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.683055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60552)
0924 01:03:46.683297 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.683412 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 01:03:46.684976 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.684534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60568)
0924 01:03:46.684705 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.684825 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 01:03:46.695363 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19974
Sep 24 01:03:46 dist-test-slave-5c7l krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:03:46.713959 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.696272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.8.254.67:35751)
0924 01:03:46.696483 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:03:46.696490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:03:46.696536 (+ 46us) server_negotiation.cc:407] Connection header received
0924 01:03:46.696605 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:03:46.696610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:03:46.696701 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:03:46.696832 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:03:46.698216 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.699495 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.701238 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:03:46.701943 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:03:46.702087 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:03:46.706594 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:03:46.706612 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:03:46.706635 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:03:46.706689 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:03:46.710799 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.711610 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.711625 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.711630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.711840 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:03:46.712206 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:03:46.712211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:03:46.712216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:03:46.712705 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:03:46.713062 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0924 01:03:46.713247 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 01:03:46.713501 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 01:03:46.714941 20097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:46.766144 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.765667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60582)
0924 01:03:46.765906 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.765993 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 01:03:46.767686 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.767185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60584)
0924 01:03:46.767433 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.767537 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260924 01:03:46.839798 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19974
I20260924 01:03:46.856837 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 19613
I20260924 01:03:46.858053 19675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:03:46.968845 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:03:46.968208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37963 (local address 127.0.0.1:60598)
0924 01:03:46.968449 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 01:03:46.968573 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 01:03:46.969143 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 19613
2026-09-24T01:03:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4354 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:03:47 dist-test-slave-5c7l krb5kdc[20154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:03:47 dist-test-slave-5c7l krb5kdc[20154](info): set up 2 sockets
Sep 24 01:03:47 dist-test-slave-5c7l krb5kdc[20154](info): commencing operation
krb5kdc: starting...
W20260924 01:03:48.080132 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:03:48 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:03:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/rangeradmin_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/rangeradmin_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/rangerlookup_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/rangerlookup_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
I20260924 01:03:48.411042 9209 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpTDkWM/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/postgres -l logfile start
2026-09-24 01:03:51.109 UTC [20184] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 01:03:51.109 UTC [20184] LOG: listening on IPv4 address "127.8.254.84", port 60339
2026-09-24 01:03:51.114 UTC [20184] LOG: listening on Unix socket "/tmp/.s.PGSQL.60339"
2026-09-24 01:03:51.138 UTC [20187] LOG: database system was shut down at 2026-09-24 01:03:49 UTC
2026-09-24 01:03:51.142 UTC [20184] LOG: database system is ready to accept connections
I20260924 01:03:52.221552 9209 mini_postgres.cc:96] Postgres bound to 60339
2026-09-24 01:03:52.251 UTC [20195] FATAL: database "slave" does not exist
127.8.254.84:60339 - accepting connections
I20260924 01:03:52.251834 9209 mini_ranger.cc:161] Starting Ranger...
I20260924 01:03:52.280646 9209 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 01:03:52.788054 9209 mini_ranger.cc:87] Created ranger Postgres database
I20260924 01:03:52.788235 9209 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-admin
2026-09-24 01:03:53,840 [I] DB FLAVOR :POSTGRES
2026-09-24 01:03:53,841 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 01:03:53,841 [I] Checking connection..
2026-09-24 01:03:53,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 01:03:54,260 [I] Checking connection passed.
2026-09-24 01:03:54,260 [I] --------- Verifying version history table ---------
2026-09-24 01:03:54,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 01:03:54,512 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 01:03:54,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 01:03:54,758 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 01:03:54,759 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 01:03:54,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 01:03:54.996 UTC [20259] WARNING: there is no transaction in progress
2026-09-24 01:03:55,004 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 01:03:55,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 01:03:55,272 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 01:03:55,273 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 01:03:55,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 01:03:55,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 01:03:55,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5c7l.c.gcp-upstream.internal','N') ;"
2026-09-24 01:03:56,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 01:03:56,271 [I] Table x_portal_user does not exist in database ranger
2026-09-24 01:03:56,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 01:03:56,528 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 01:03:56,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 01:03:56,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 01:03:57,031 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 01:03:57,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 01:03:58.015 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.031 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.047 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.063 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.079 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.099 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.156 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.163 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.179 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.190 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.200 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.214 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.224 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.231 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.239 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58.246 UTC [20367] WARNING: there is no transaction in progress
2026-09-24 01:03:58,896 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 01:03:58,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 01:03:59,133 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 01:03:59,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 01:03:59,394 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 01:03:59,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 01:03:59,664 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 01:03:59,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 01:03:59,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 01:04:00,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5c7l.c.gcp-upstream.internal';"
2026-09-24 01:04:00,380 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 01:04:00,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 01:04:00,616 [I] DB_PATCHES have already been applied
I20260924 01:04:00.622123 9209 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpTDkWM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 01:04:00.622320 9209 mini_ranger.cc:193] Using host: 127.8.254.84
I20260924 01:04:00.638075 9209 mini_ranger.cc:227] Ranger admin URL: http://127.8.254.84:51275
Sep 24, 2026 1:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:04:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:04:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:04:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 01:04:02.897156 9209 mini_ranger.cc:160] Time spent starting Ranger: real 10.645s user 0.000s sys 0.051s
Sep 24, 2026 1:04:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:04:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 1:04:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:04:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:04:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 01:04:26.382854 9209 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/rangerkms_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/rangerkms_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.254.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.254.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 01:04:26 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 01:04:26.551636 9209 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 01:04:26.574685 9209 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 01:04:27.500144 9209 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 01:04:27.500279 9209 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms
I20260924 01:04:27.500427 9209 mini_ranger_kms.cc:227] Using postgres at 127.8.254.84:60339
2026-09-24 01:04:28,386 [I] DB FLAVOR :POSTGRES
2026-09-24 01:04:28,387 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 01:04:28,387 [I] Checking connection
2026-09-24 01:04:28,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 01:04:28,653 [I] connection success
2026-09-24 01:04:28,653 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 01:04:28,653 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 01:04:28,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 01:04:28,892 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 01:04:28,893 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 01:04:28,893 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 01:04:28,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:60339/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 01:04:29,187 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 01:04:29.709918 9209 mini_ranger_kms.cc:310] Created kms service
I20260924 01:04:29.886987 9209 mini_ranger_kms.cc:326] Created kudu user
I20260924 01:04:29.935052 9209 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 01:04:29 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:04:30.367797 9209 mini_ranger_kms.cc:384] Added ranger policy
I20260924 01:04:30.368139 9209 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 01:04:30.368204 9209 mini_ranger_kms.cc:240] Using host: 127.8.254.84
I20260924 01:04:30.389503 9209 mini_ranger_kms.cc:276] Ranger KMS PID: 20549
I20260924 01:04:30.389716 9209 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.254.84:41607
Sep 24, 2026 1:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 01:04:32 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.254.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:04:32.768784 9209 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.217s user 0.138s sys 0.430s
I20260924 01:04:32.769049 9209 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 01:04:32.769287 9209 mini_ranger_kms.cc:401] 127.8.254.84:41607/kms/v1/keys
Sep 24 01:04:33 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.254.84@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
Sep 24 01:04:33 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.254.84@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
Sep 24 01:04:35 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:04:36.014117 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:33729
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:33729
--ranger_config_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:41607/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:04:36.392750 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:04:36.393050 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:04:36.393138 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:04:36.400653 20599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:04:36.400776 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:04:36.400823 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:04:36.400884 20599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:04:36.400935 20599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:04:36.411226 20599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.254.84:41607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:33729
--ranger_config_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:04:36.412565 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:04:36.413955 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:04:36.426311 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:36.426227 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:36.426357 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:04:36.426702 20599 server_base.cc:1034] running on GCE node
I20260924 01:04:36.427635 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:04:36.429071 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:04:36.430294 20599 hybrid_clock.cc:648] HybridClock initialized: now 1790211876430240 us; error 56 us; skew 500 ppm
I20260924 01:04:36.430702 20599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:04:36 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:04:36.446811 20599 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:04:36.448665 20599 webserver.cc:460] Webserver started at http://127.8.254.126:42181/ using document root <none> and password file <none>
I20260924 01:04:36.449379 20599 fs_manager.cc:359] Metadata directory not provided
I20260924 01:04:36.449482 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:04:36.449757 20599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:04:36 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:04:36.507256 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "bb8eed65d555466ca2cbc2f964a1701b"
format_stamp: "Formatted at 2026-09-24 01:04:36 on dist-test-slave-5c7l"
server_key: "5ec8e8b9e8342a08099a0275eee271ef"
server_key_iv: "63106c91bae490dd6865f65979232c09"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:36.508275 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "bb8eed65d555466ca2cbc2f964a1701b"
format_stamp: "Formatted at 2026-09-24 01:04:36 on dist-test-slave-5c7l"
server_key: "5ec8e8b9e8342a08099a0275eee271ef"
server_key_iv: "63106c91bae490dd6865f65979232c09"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:36.515334 20599 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260924 01:04:36.538841 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:36.540709 20599 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.008s
I20260924 01:04:36.540908 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "bb8eed65d555466ca2cbc2f964a1701b"
format_stamp: "Formatted at 2026-09-24 01:04:36 on dist-test-slave-5c7l"
server_key: "5ec8e8b9e8342a08099a0275eee271ef"
server_key_iv: "63106c91bae490dd6865f65979232c09"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:36.541163 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:04:36.558032 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:04:36.561792 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:04:36.562126 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:04:36.581266 20599 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:33729
I20260924 01:04:36.581301 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:33729 every 8 connection(s)
I20260924 01:04:36.582676 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:04:36.583259 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 20599
I20260924 01:04:36.583467 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:04:36.584296 9209 external_mini_cluster.cc:1428] Setting key 74e2c293c21e002223b0285fc4c85bc5
I20260924 01:04:36.587245 20666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 01:04:36 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:04:36.597963 20666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b: Bootstrap starting.
I20260924 01:04:36.601150 20666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b: Neither blocks nor log segments found. Creating new log.
I20260924 01:04:36.602241 20666 log.cc:824] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b: Log is configured to *not* fsync() on all Append() calls
I20260924 01:04:36.605129 20666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b: No bootstrap required, opened a new log
I20260924 01:04:36.606048 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:36.586348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.8.254.126:33729)
0924 01:04:36.587116 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:36.587129 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:36.587168 (+ 39us) server_negotiation.cc:407] Connection header received
0924 01:04:36.588817 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:36.588839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:36.589226 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:36.589647 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:36.591044 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:36.592458 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:36.593734 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:36.594498 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:36.594699 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:36.598833 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:36.598841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:36.598856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:36.598892 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:36.601964 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:36.602775 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:36.602782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:36.602788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:36.602955 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:36.603369 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:36.603372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:36.603378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:36.603859 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:36.604154 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:36.604615 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 01:04:36.605010 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":327,"threads_started":1}
I20260924 01:04:36.613232 20666 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8eed65d555466ca2cbc2f964a1701b" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33729 } }
I20260924 01:04:36.613631 20666 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:04:36.613739 20666 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8eed65d555466ca2cbc2f964a1701b, State: Initialized, Role: FOLLOWER
I20260924 01:04:36.614321 20666 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8eed65d555466ca2cbc2f964a1701b" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33729 } }
I20260924 01:04:36.614570 20666 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:04:36.614710 20666 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:04:36.614880 20666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:04:36.617318 20666 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8eed65d555466ca2cbc2f964a1701b" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33729 } }
I20260924 01:04:36.617805 20666 leader_election.cc:302] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb8eed65d555466ca2cbc2f964a1701b; no voters:
I20260924 01:04:36.618432 20666 leader_election.cc:288] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:04:36.618567 20671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:04:36.619429 20671 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [term 1 LEADER]: Becoming Leader. State: Replica: bb8eed65d555466ca2cbc2f964a1701b, State: Running, Role: LEADER
I20260924 01:04:36.620008 20671 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8eed65d555466ca2cbc2f964a1701b" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33729 } }
I20260924 01:04:36.620653 20666 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:04:36.623701 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb8eed65d555466ca2cbc2f964a1701b. Latest consensus state: current_term: 1 leader_uuid: "bb8eed65d555466ca2cbc2f964a1701b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8eed65d555466ca2cbc2f964a1701b" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33729 } } }
I20260924 01:04:36.624200 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [sys.catalog]: This master's current role is: LEADER
I20260924 01:04:36.625005 20672 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb8eed65d555466ca2cbc2f964a1701b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8eed65d555466ca2cbc2f964a1701b" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33729 } } }
I20260924 01:04:36.625378 20672 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b [sys.catalog]: This master's current role is: LEADER
I20260924 01:04:36.626046 20681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:04:36.631307 20681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:04:36.632051 20666 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 01:04:36.659446 20681 catalog_manager.cc:1347] Generated new cluster ID: c5a3182db896493392c5f078af8191c4
I20260924 01:04:36.659622 20681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:04:36.684706 20681 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:04:36.686381 20681 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:04:36.702921 20681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b: Generated new TSK 0
I20260924 01:04:36.704187 20681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:04:37.483481 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:41607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:33729
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:04:37 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 01:04:37.919188 20711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:04:37.919512 20711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:04:37.919737 20711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:04:37.928093 20711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:04:37.928259 20711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:04:37.928432 20711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:04:37.941156 20711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.254.84:41607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:04:37.942623 20711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:04:37.943936 20711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:04:37.962664 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:37.963022 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:37.968811 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:04:37.969286 20711 server_base.cc:1034] running on GCE node
I20260924 01:04:37.969990 20711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:04:37.971292 20711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:04:37.972587 20711 hybrid_clock.cc:648] HybridClock initialized: now 1790211877972497 us; error 70 us; skew 500 ppm
I20260924 01:04:37.972941 20711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:04:37 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:04:37.976934 20711 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:04:37.999126 20711 webserver.cc:460] Webserver started at http://127.8.254.65:33613/ using document root <none> and password file <none>
I20260924 01:04:38.000406 20711 fs_manager.cc:359] Metadata directory not provided
I20260924 01:04:38.000586 20711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:04:38.000936 20711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:04:38 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:04:38.018733 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "7fb9a702ba324f618778c901239e3e1d"
format_stamp: "Formatted at 2026-09-24 01:04:38 on dist-test-slave-5c7l"
server_key: "1182816178cc576b6bdd5855448ab22d"
server_key_iv: "b8ca05a8c26842ddfd01f3b7e6a5edd7"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:38.020216 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "7fb9a702ba324f618778c901239e3e1d"
format_stamp: "Formatted at 2026-09-24 01:04:38 on dist-test-slave-5c7l"
server_key: "1182816178cc576b6bdd5855448ab22d"
server_key_iv: "b8ca05a8c26842ddfd01f3b7e6a5edd7"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:38.044639 20711 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.007s sys 0.015s
I20260924 01:04:38.083119 20731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:38.085310 20711 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.006s
I20260924 01:04:38.085596 20711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "7fb9a702ba324f618778c901239e3e1d"
format_stamp: "Formatted at 2026-09-24 01:04:38 on dist-test-slave-5c7l"
server_key: "1182816178cc576b6bdd5855448ab22d"
server_key_iv: "b8ca05a8c26842ddfd01f3b7e6a5edd7"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:38.085966 20711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:04:38.101586 20711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:04:38.105953 20711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:04:38.106333 20711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:04:38.107610 20711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:04:38.109449 20711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:04:38.109606 20711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:38.109802 20711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:04:38.109881 20711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:38.188877 20711 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:39989
I20260924 01:04:38.190909 20711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:04:38.188901 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:39989 every 8 connection(s)
I20260924 01:04:38.194474 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 20711
I20260924 01:04:38.194700 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:04:38.195561 9209 external_mini_cluster.cc:1428] Setting key 3ba8ab4b52e67d4141f7727f6ea09807
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Sep 24 01:04:38 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:04:38.264099 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:38.196958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:35759 (local address 127.8.254.126:33729)
0924 01:04:38.223613 (+ 26655us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:38.223622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:38.227088 (+ 3466us) server_negotiation.cc:407] Connection header received
0924 01:04:38.236124 (+ 9036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:38.236132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:38.236256 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:38.236344 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:38.239668 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:38.241032 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:38.243716 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:38.244362 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:38.244461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:38.254007 (+ 9546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:38.254015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:38.254042 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:38.254092 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:38.257772 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:38.260095 (+ 2323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:38.260102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:38.260106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:38.260282 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:38.260870 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:38.260875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:38.260878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:38.261233 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:38.261361 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:38.263434 (+ 2073us) server_negotiation.cc:299] Negotiation successful
0924 01:04:38.263692 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26487,"thread_start_us":26373,"threads_started":1}
I20260924 01:04:38.267750 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:38.196145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:33729 (local address 127.8.254.65:35759)
0924 01:04:38.224910 (+ 28765us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:38.224978 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:38.235624 (+ 10646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:38.237205 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:38.237221 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:38.238061 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:04:38.239341 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:38.239372 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:38.241300 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:38.241307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:38.243531 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:38.243547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:38.244892 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:38.244901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:38.245219 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:38.249934 (+ 4715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:04:38.249979 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:04:38.253263 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:04:38.258339 (+ 5076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:04:38.258349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:04:38.258371 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:04:38.259004 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:04:38.260449 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:04:38.260453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:04:38.260457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:04:38.260736 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:04:38.261458 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:04:38.261467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:04:38.261941 (+ 474us) client_negotiation.cc:769] Sending connection context
0924 01:04:38.265622 (+ 3681us) client_negotiation.cc:240] Negotiation successful
0924 01:04:38.266256 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28392,"thread_start_us":28584,"threads_started":1}
I20260924 01:04:38.270007 20844 heartbeater.cc:347] Connected to a master server at 127.8.254.126:33729
I20260924 01:04:38.270380 20844 heartbeater.cc:464] Registering TS with master...
I20260924 01:04:38.271324 20844 heartbeater.cc:511] Master 127.8.254.126:33729 requested a full tablet report, sending...
I20260924 01:04:38.274116 20631 ts_manager.cc:194] Registered new tserver with Master: 7fb9a702ba324f618778c901239e3e1d (127.8.254.65:39989)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:04:38.276676 20631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:35759
I20260924 01:04:38.278257 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:41607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:33729
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:04:38 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
Sep 24 01:04:38 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
W20260924 01:04:38.777338 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:04:38.777890 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:04:38.778103 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:04:38.789548 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:04:38.789785 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:04:38.790006 20850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:04:38.805655 20850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.254.84:41607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:04:38.807758 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:04:38.809572 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:04:38.838960 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:38.840715 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:04:38.849076 20850 server_base.cc:1034] running on GCE node
W20260924 01:04:38.849512 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:04:38.850458 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:04:38.854498 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:04:38.855836 20850 hybrid_clock.cc:648] HybridClock initialized: now 1790211878855721 us; error 100 us; skew 500 ppm
I20260924 01:04:38.856269 20850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:04:38 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:04:38.860744 20850 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:04:38.874855 20850 webserver.cc:460] Webserver started at http://127.8.254.66:41899/ using document root <none> and password file <none>
I20260924 01:04:38.875926 20850 fs_manager.cc:359] Metadata directory not provided
I20260924 01:04:38.876082 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:04:38.876487 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:04:38 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:04:38.904323 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "de0253d4ad924b399309e23f04717c24"
format_stamp: "Formatted at 2026-09-24 01:04:38 on dist-test-slave-5c7l"
server_key: "64ed0e23e5f9780b558896271a245079"
server_key_iv: "48748328997845717108fb94f8fd1051"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:38.906390 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "de0253d4ad924b399309e23f04717c24"
format_stamp: "Formatted at 2026-09-24 01:04:38 on dist-test-slave-5c7l"
server_key: "64ed0e23e5f9780b558896271a245079"
server_key_iv: "48748328997845717108fb94f8fd1051"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:38.914001 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 01:04:38.945940 20867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:38.952061 20850 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.009s
I20260924 01:04:38.952386 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "de0253d4ad924b399309e23f04717c24"
format_stamp: "Formatted at 2026-09-24 01:04:38 on dist-test-slave-5c7l"
server_key: "64ed0e23e5f9780b558896271a245079"
server_key_iv: "48748328997845717108fb94f8fd1051"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:38.952762 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:04:39.008347 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:04:39.013793 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:04:39.014137 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:04:39.015359 20850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:04:39.017102 20850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:04:39.017196 20850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:39.017355 20850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:04:39.017414 20850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:39.065454 20850 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:34125
I20260924 01:04:39.065588 20986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:34125 every 8 connection(s)
I20260924 01:04:39.067131 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:04:39.069926 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 20850
I20260924 01:04:39.070111 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:04:39.070693 9209 external_mini_cluster.cc:1428] Setting key 4ec72409cfd352217fa2bc0d300e7a53
Sep 24 01:04:39 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:04:39.097645 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.074086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:55259 (local address 127.8.254.126:33729)
0924 01:04:39.075724 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.075732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.078287 (+ 2555us) server_negotiation.cc:407] Connection header received
0924 01:04:39.078494 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.078499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.078588 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.078700 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:39.080484 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.081857 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.084246 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.084986 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.085139 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.089599 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:39.089606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:39.089611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:39.089654 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:39.093913 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:39.094904 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:39.094910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:39.094914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:39.095076 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:39.095636 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:39.095644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:39.095649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:39.096048 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:39.096221 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.097117 (+ 896us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.097354 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1435,"threads_started":1}
I20260924 01:04:39.098923 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.073438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:33729 (local address 127.8.254.66:55259)
0924 01:04:39.076298 (+ 2860us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.076346 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.078133 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.079023 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.079034 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.079475 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:04:39.080248 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.080275 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.082075 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.082082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.084071 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.084086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.085253 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.085259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.085534 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.086379 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:04:39.086407 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:04:39.089340 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:04:39.094099 (+ 4759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:04:39.094108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:04:39.094129 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:04:39.094744 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:04:39.095208 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:04:39.095213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:04:39.095217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:04:39.095471 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:04:39.096195 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:04:39.096204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:04:39.096647 (+ 443us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.096941 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.097493 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2530,"thread_start_us":2289,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
I20260924 01:04:39.107034 20987 heartbeater.cc:347] Connected to a master server at 127.8.254.126:33729
I20260924 01:04:39.107438 20987 heartbeater.cc:464] Registering TS with master...
I20260924 01:04:39.108316 20987 heartbeater.cc:511] Master 127.8.254.126:33729 requested a full tablet report, sending...
I20260924 01:04:39.110169 20631 ts_manager.cc:194] Registered new tserver with Master: de0253d4ad924b399309e23f04717c24 (127.8.254.66:34125)
I20260924 01:04:39.111361 20631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:55259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:04:39.131263 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:41607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:33729
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:04:39.281092 20844 heartbeater.cc:503] Master 127.8.254.126:33729 was elected leader, sending a full tablet report...
W20260924 01:04:39.478529 20993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:04:39.478849 20993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:04:39.479014 20993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:04:39.486410 20993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:04:39.486549 20993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:04:39.486757 20993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:04:39.502802 20993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.254.84:41607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:33729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:04:39.504776 20993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:04:39.506330 20993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:04:39.521045 20999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:39.521147 20998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:04:39.521598 21001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:04:39.521955 20993 server_base.cc:1034] running on GCE node
I20260924 01:04:39.522710 20993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:04:39.523785 20993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:04:39.525010 20993 hybrid_clock.cc:648] HybridClock initialized: now 1790211879524942 us; error 62 us; skew 500 ppm
I20260924 01:04:39.525346 20993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:04:39 dist-test-slave-5c7l krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:04:39.528646 20993 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:04:39.530622 20993 webserver.cc:460] Webserver started at http://127.8.254.67:36963/ using document root <none> and password file <none>
I20260924 01:04:39.531347 20993 fs_manager.cc:359] Metadata directory not provided
I20260924 01:04:39.531476 20993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:04:39.531829 20993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:04:39 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:04:39.549078 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "51564de99bae4d53bf18ff98d2d72f96"
format_stamp: "Formatted at 2026-09-24 01:04:39 on dist-test-slave-5c7l"
server_key: "e0a9bc90165a2fb202bd6e7d02662f93"
server_key_iv: "541fdb4085a46f4bde38ba92ff04dea6"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:39.550259 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "51564de99bae4d53bf18ff98d2d72f96"
format_stamp: "Formatted at 2026-09-24 01:04:39 on dist-test-slave-5c7l"
server_key: "e0a9bc90165a2fb202bd6e7d02662f93"
server_key_iv: "541fdb4085a46f4bde38ba92ff04dea6"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:39.558401 20993 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260924 01:04:39.575795 21008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:39.577757 20993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 01:04:39.577963 20993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "51564de99bae4d53bf18ff98d2d72f96"
format_stamp: "Formatted at 2026-09-24 01:04:39 on dist-test-slave-5c7l"
server_key: "e0a9bc90165a2fb202bd6e7d02662f93"
server_key_iv: "541fdb4085a46f4bde38ba92ff04dea6"
server_key_version: "kuduclusterkey@0"
I20260924 01:04:39.578203 20993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:04:39.601212 20993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:04:39.605477 20993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:04:39.605808 20993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:04:39.606935 20993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:04:39.608374 20993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:04:39.608480 20993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:39.608631 20993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:04:39.608704 20993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:04:39.643424 20993 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:37449
I20260924 01:04:39.643460 21120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:37449 every 8 connection(s)
I20260924 01:04:39.644871 20993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:04:39.648201 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 20993
I20260924 01:04:39.648427 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:04:39.648871 9209 external_mini_cluster.cc:1428] Setting key ca8396ba3c70059828974457284c05b9
Sep 24 01:04:39 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:04:39.667269 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.647551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:38421 (local address 127.8.254.126:33729)
0924 01:04:39.648202 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.648208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.649343 (+ 1135us) server_negotiation.cc:407] Connection header received
0924 01:04:39.651469 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.651476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.651565 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.651687 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:39.653522 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.654335 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.655987 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.656421 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.656527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.660465 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:39.660470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:39.660474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:39.660512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:39.663536 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:39.664663 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:39.664668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:39.664671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:39.664850 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:39.665522 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:39.665526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:39.665528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:39.665876 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:39.666012 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.666820 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.667009 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":317,"threads_started":1}
I20260924 01:04:39.668315 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.648241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:33729 (local address 127.8.254.67:38421)
0924 01:04:39.649150 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.649206 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.651197 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.652071 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.652081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.652660 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:04:39.653328 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.653349 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.654505 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.654509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.655837 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.655850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.656720 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.656725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.656958 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.657756 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:04:39.657781 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:04:39.660194 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:04:39.663732 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:04:39.663747 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:04:39.663770 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:04:39.664529 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:04:39.665030 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:04:39.665036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:04:39.665040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:04:39.665366 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:04:39.666026 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:04:39.666033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:04:39.666477 (+ 444us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.666736 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.667169 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":407,"threads_started":1}
I20260924 01:04:39.669837 21121 heartbeater.cc:347] Connected to a master server at 127.8.254.126:33729
I20260924 01:04:39.670123 21121 heartbeater.cc:464] Registering TS with master...
I20260924 01:04:39.670877 21121 heartbeater.cc:511] Master 127.8.254.126:33729 requested a full tablet report, sending...
I20260924 01:04:39.672339 20631 ts_manager.cc:194] Registered new tserver with Master: 51564de99bae4d53bf18ff98d2d72f96 (127.8.254.67:37449)
I20260924 01:04:39.673174 20631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:38421
I20260924 01:04:39.677583 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:04:39.695641 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.682107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.8.254.126:33729)
0924 01:04:39.682322 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.682326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.682844 (+ 518us) server_negotiation.cc:407] Connection header received
0924 01:04:39.683054 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.683060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.683153 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.683262 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:39.684743 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.685532 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.687037 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.687463 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.687598 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.690038 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:39.690043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:39.690046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:39.690078 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:39.692779 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:39.693579 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:39.693586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:39.693590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:39.693725 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:39.694104 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:39.694110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:39.694113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:39.694462 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:39.694715 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.695079 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.695321 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:04:39.699952 20631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:04:39.707166 20631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:04:39.746874 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.720779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.8.254.65:39989)
0924 01:04:39.737272 (+ 16493us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.737284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.738110 (+ 826us) server_negotiation.cc:407] Connection header received
0924 01:04:39.738510 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.738520 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.738823 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.739015 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:04:39.740215 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.741447 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.744164 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.745612 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.745774 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.746035 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.746340 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.746509 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16285,"thread_start_us":16135,"threads_started":1}
I20260924 01:04:39.746940 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.720160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39989 (local address 127.0.0.1:55512)
0924 01:04:39.737975 (+ 17815us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.737999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.738263 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.739457 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.739464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.739497 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:04:39.739958 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.739975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.741593 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.741601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.743959 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.743983 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.745816 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.745822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.746016 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.746084 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.746292 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.746493 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17676,"mutex_wait_us":4,"thread_start_us":18221,"threads_started":1}
I20260924 01:04:39.747813 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.719999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:34125 (local address 127.0.0.1:38066)
0924 01:04:39.735296 (+ 15297us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.735359 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.735603 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.740067 (+ 4464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.740072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.740120 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:04:39.740511 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.740520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.742665 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.742670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.744997 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.745010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.746956 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.746964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.747139 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.747188 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.747260 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.747395 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14894,"mutex_wait_us":5,"spinlock_wait_cycles":6528,"thread_start_us":15866,"threads_started":1}
I20260924 01:04:39.748152 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.723274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.8.254.67:37449)
0924 01:04:39.739232 (+ 15958us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.739241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.739278 (+ 37us) server_negotiation.cc:407] Connection header received
0924 01:04:39.739378 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.739383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.739576 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.739793 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:04:39.740701 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.742391 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.745231 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.746706 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.746848 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.747091 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.747638 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.747804 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15627,"thread_start_us":15465,"threads_started":1}
I20260924 01:04:39.748754 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.720173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.8.254.66:34125)
0924 01:04:39.739227 (+ 19054us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.739239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.739364 (+ 125us) server_negotiation.cc:407] Connection header received
0924 01:04:39.739469 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.739477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.739836 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.740735 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:04:39.740973 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.742464 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.745153 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.746706 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.747941 (+ 1235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.748157 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.748268 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.748437 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18882,"thread_start_us":18942,"threads_started":1}
I20260924 01:04:39.749697 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.722972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:37449 (local address 127.0.0.1:47314)
0924 01:04:39.738613 (+ 15641us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.738630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.738722 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.739781 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.739785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.739812 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:04:39.740480 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.740498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.742550 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.742556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.745003 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.745017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.747259 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.747267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.747436 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.747465 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.747566 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.747689 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15542,"mutex_wait_us":108,"thread_start_us":14730,"threads_started":1}
I20260924 01:04:39.751590 20779 tablet_service.cc:1467] Processing CreateTablet for tablet 0b618bf0d4044e66ae16bb15c322b210 (DEFAULT_TABLE table=test-table [id=de1c6475c19f454c8ed237343f002413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:04:39.752158 20922 tablet_service.cc:1467] Processing CreateTablet for tablet 0b618bf0d4044e66ae16bb15c322b210 (DEFAULT_TABLE table=test-table [id=de1c6475c19f454c8ed237343f002413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:04:39.753247 20779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b618bf0d4044e66ae16bb15c322b210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:04:39.753059 21056 tablet_service.cc:1467] Processing CreateTablet for tablet 0b618bf0d4044e66ae16bb15c322b210 (DEFAULT_TABLE table=test-table [id=de1c6475c19f454c8ed237343f002413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:04:39.753543 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b618bf0d4044e66ae16bb15c322b210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:04:39.754347 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b618bf0d4044e66ae16bb15c322b210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:04:39.766271 21140 tablet_bootstrap.cc:492] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Bootstrap starting.
I20260924 01:04:39.768504 21142 tablet_bootstrap.cc:492] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Bootstrap starting.
I20260924 01:04:39.770485 21140 tablet_bootstrap.cc:654] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Neither blocks nor log segments found. Creating new log.
I20260924 01:04:39.771911 21140 log.cc:824] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Log is configured to *not* fsync() on all Append() calls
I20260924 01:04:39.772521 21142 tablet_bootstrap.cc:654] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Neither blocks nor log segments found. Creating new log.
I20260924 01:04:39.773332 21141 tablet_bootstrap.cc:492] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Bootstrap starting.
I20260924 01:04:39.775234 21142 log.cc:824] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Log is configured to *not* fsync() on all Append() calls
I20260924 01:04:39.775560 21140 tablet_bootstrap.cc:492] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: No bootstrap required, opened a new log
I20260924 01:04:39.775856 21140 ts_tablet_manager.cc:1397] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260924 01:04:39.777086 21141 tablet_bootstrap.cc:654] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Neither blocks nor log segments found. Creating new log.
I20260924 01:04:39.778347 21141 log.cc:824] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Log is configured to *not* fsync() on all Append() calls
I20260924 01:04:39.781608 21142 tablet_bootstrap.cc:492] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: No bootstrap required, opened a new log
I20260924 01:04:39.781611 21141 tablet_bootstrap.cc:492] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: No bootstrap required, opened a new log
I20260924 01:04:39.781899 21141 ts_tablet_manager.cc:1397] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 01:04:39.781899 21142 ts_tablet_manager.cc:1397] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260924 01:04:39.785415 21140 raft_consensus.cc:348] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.785808 21140 raft_consensus.cc:374] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:04:39.785939 21140 raft_consensus.cc:729] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51564de99bae4d53bf18ff98d2d72f96, State: Initialized, Role: FOLLOWER
I20260924 01:04:39.786633 21140 consensus_queue.cc:260] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.787740 21121 heartbeater.cc:503] Master 127.8.254.126:33729 was elected leader, sending a full tablet report...
I20260924 01:04:39.788477 21140 ts_tablet_manager.cc:1428] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 01:04:39.793097 21142 raft_consensus.cc:348] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.793028 21141 raft_consensus.cc:348] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.793556 21141 raft_consensus.cc:374] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:04:39.793555 21142 raft_consensus.cc:374] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:04:39.793668 21142 raft_consensus.cc:729] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de0253d4ad924b399309e23f04717c24, State: Initialized, Role: FOLLOWER
I20260924 01:04:39.793668 21141 raft_consensus.cc:729] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb9a702ba324f618778c901239e3e1d, State: Initialized, Role: FOLLOWER
I20260924 01:04:39.794350 21141 consensus_queue.cc:260] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.794348 21142 consensus_queue.cc:260] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.795944 20987 heartbeater.cc:503] Master 127.8.254.126:33729 was elected leader, sending a full tablet report...
I20260924 01:04:39.796345 21141 ts_tablet_manager.cc:1428] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260924 01:04:39.796522 21142 ts_tablet_manager.cc:1428] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
W20260924 01:04:39.823408 20988 tablet.cc:2369] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:04:39.869032 21147 raft_consensus.cc:482] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:04:39.869351 21147 raft_consensus.cc:504] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.871145 21147 leader_election.cc:288] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb9a702ba324f618778c901239e3e1d (127.8.254.65:39989), 51564de99bae4d53bf18ff98d2d72f96 (127.8.254.67:37449)
I20260924 01:04:39.878515 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.871727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:55363 (local address 127.8.254.65:39989)
0924 01:04:39.871979 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.871987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.872746 (+ 759us) server_negotiation.cc:407] Connection header received
0924 01:04:39.872915 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.872921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.873005 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.873088 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:04:39.873779 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.874730 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.876648 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.877447 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.877636 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.877714 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.878044 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.878188 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 01:04:39.878616 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.871725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:56571 (local address 127.8.254.67:37449)
0924 01:04:39.871916 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.871920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.872882 (+ 962us) server_negotiation.cc:407] Connection header received
0924 01:04:39.873042 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.873046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.873163 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.873262 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:04:39.873920 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.874956 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.876825 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.877605 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.877802 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.877885 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.878232 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.878360 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:04:39.878671 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.871671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:39989 (local address 127.8.254.66:55363)
0924 01:04:39.872595 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.872616 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.872771 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.873131 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.873136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.873160 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:04:39.873611 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.873622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.874863 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.874868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.876483 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.876495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.877670 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.877677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.877833 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.877896 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.878008 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.878140 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":19,"spinlock_wait_cycles":6912,"thread_start_us":732,"threads_started":1}
I20260924 01:04:39.878774 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.871671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:37449 (local address 127.8.254.66:56571)
0924 01:04:39.872726 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 01:04:39.872758 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 01:04:39.872915 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:04:39.873295 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:04:39.873298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:04:39.873340 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:04:39.873728 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.873740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.875117 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.875123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.876643 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:04:39.876655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.877824 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.877831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:04:39.878051 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.878074 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 01:04:39.878181 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 01:04:39.878338 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":61,"spinlock_wait_cycles":2944,"thread_start_us":464,"threads_started":1}
I20260924 01:04:39.879928 21076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b618bf0d4044e66ae16bb15c322b210" candidate_uuid: "de0253d4ad924b399309e23f04717c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51564de99bae4d53bf18ff98d2d72f96" is_pre_election: true
I20260924 01:04:39.879966 20799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b618bf0d4044e66ae16bb15c322b210" candidate_uuid: "de0253d4ad924b399309e23f04717c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb9a702ba324f618778c901239e3e1d" is_pre_election: true
I20260924 01:04:39.880559 21076 raft_consensus.cc:2454] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de0253d4ad924b399309e23f04717c24 in term 0.
I20260924 01:04:39.880559 20799 raft_consensus.cc:2454] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de0253d4ad924b399309e23f04717c24 in term 0.
I20260924 01:04:39.881280 20871 leader_election.cc:302] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51564de99bae4d53bf18ff98d2d72f96, 7fb9a702ba324f618778c901239e3e1d, de0253d4ad924b399309e23f04717c24; no voters:
I20260924 01:04:39.881700 21147 raft_consensus.cc:2781] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:04:39.881891 21147 raft_consensus.cc:482] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:04:39.882017 21147 raft_consensus.cc:3037] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:04:39.884444 21147 raft_consensus.cc:504] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
I20260924 01:04:39.885051 21147 leader_election.cc:288] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [CANDIDATE]: Term 1 election: Requested vote from peers 7fb9a702ba324f618778c901239e3e1d (127.8.254.65:39989), 51564de99bae4d53bf18ff98d2d72f96 (127.8.254.67:37449)
I20260924 01:04:39.885604 21076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b618bf0d4044e66ae16bb15c322b210" candidate_uuid: "de0253d4ad924b399309e23f04717c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51564de99bae4d53bf18ff98d2d72f96"
I20260924 01:04:39.885604 20799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b618bf0d4044e66ae16bb15c322b210" candidate_uuid: "de0253d4ad924b399309e23f04717c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb9a702ba324f618778c901239e3e1d"
I20260924 01:04:39.885831 21076 raft_consensus.cc:3037] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:04:39.885831 20799 raft_consensus.cc:3037] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:04:39.889114 21076 raft_consensus.cc:2454] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de0253d4ad924b399309e23f04717c24 in term 1.
I20260924 01:04:39.889114 20799 raft_consensus.cc:2454] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de0253d4ad924b399309e23f04717c24 in term 1.
I20260924 01:04:39.889599 20871 leader_election.cc:302] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51564de99bae4d53bf18ff98d2d72f96, 7fb9a702ba324f618778c901239e3e1d, de0253d4ad924b399309e23f04717c24; no voters:
I20260924 01:04:39.889956 21147 raft_consensus.cc:2781] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:04:39.890573 21147 raft_consensus.cc:686] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 1 LEADER]: Becoming Leader. State: Replica: de0253d4ad924b399309e23f04717c24, State: Running, Role: LEADER
I20260924 01:04:39.891116 21147 consensus_queue.cc:237] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } }
W20260924 01:04:39.896605 21122 tablet.cc:2369] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:04:39.896413 20631 catalog_manager.cc:5495] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to de0253d4ad924b399309e23f04717c24 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "de0253d4ad924b399309e23f04717c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de0253d4ad924b399309e23f04717c24" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 34125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 } health_report { overall_health: UNKNOWN } } }
I20260924 01:04:39.924165 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.918931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.8.254.66:34125)
0924 01:04:39.919095 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.919099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.919117 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:04:39.919202 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.919206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.919287 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.919419 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:04:39.919818 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.920634 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.922078 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.922507 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:39.922605 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:39.923123 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:04:39.923624 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:04:39.923750 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:39.923821 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 01:04:39.923929 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 01:04:39.932854 21076 raft_consensus.cc:1261] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 1 FOLLOWER]: Refusing update from remote peer de0253d4ad924b399309e23f04717c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:04:39.932854 20799 raft_consensus.cc:1261] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 1 FOLLOWER]: Refusing update from remote peer de0253d4ad924b399309e23f04717c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:04:39.933908 21151 consensus_queue.cc:1035] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51564de99bae4d53bf18ff98d2d72f96" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:04:39.934296 21147 consensus_queue.cc:1035] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb9a702ba324f618778c901239e3e1d" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:04:39.950233 21156 mvcc.cc:205] Tried to move back new op lower bound from 7332707860194762752 to 7332707860037967872. Current Snapshot: MvccSnapshot[applied={T|T < 7332707860194762752}]
I20260924 01:04:39.964875 20630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:04:39.965267 20630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:04:39.969151 20630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb8eed65d555466ca2cbc2f964a1701b: Sending DeleteTablet for 3 replicas of tablet 0b618bf0d4044e66ae16bb15c322b210
I20260924 01:04:39.970383 20922 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b618bf0d4044e66ae16bb15c322b210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:04:39 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:38066
I20260924 01:04:39.970481 20779 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b618bf0d4044e66ae16bb15c322b210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:04:39 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:55512
I20260924 01:04:39.970575 21056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b618bf0d4044e66ae16bb15c322b210 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:04:39 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:47314
I20260924 01:04:39.973742 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 20711
I20260924 01:04:39.980448 21167 tablet_replica.cc:330] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: stopping tablet replica
I20260924 01:04:39.980718 21165 tablet_replica.cc:330] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: stopping tablet replica
I20260924 01:04:39.981151 21167 raft_consensus.cc:2229] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:04:39.981196 21165 raft_consensus.cc:2229] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:04:39.981796 21167 raft_consensus.cc:2258] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:04:39.981997 21165 raft_consensus.cc:2258] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:04:39.984414 21165 ts_tablet_manager.cc:1905] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:04:39.984406 21167 ts_tablet_manager.cc:1905] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:04:39.992555 21167 ts_tablet_manager.cc:1918] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:04:39.992823 21167 log.cc:1196] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/0b618bf0d4044e66ae16bb15c322b210
I20260924 01:04:39.993384 21167 ts_tablet_manager.cc:1939] T 0b618bf0d4044e66ae16bb15c322b210 P 51564de99bae4d53bf18ff98d2d72f96: Deleting consensus metadata
I20260924 01:04:39.994052 21165 ts_tablet_manager.cc:1918] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:04:39.994372 21165 log.cc:1196] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/0b618bf0d4044e66ae16bb15c322b210
I20260924 01:04:39.995033 21165 ts_tablet_manager.cc:1939] T 0b618bf0d4044e66ae16bb15c322b210 P de0253d4ad924b399309e23f04717c24: Deleting consensus metadata
I20260924 01:04:39.995839 20615 catalog_manager.cc:4841] TS 51564de99bae4d53bf18ff98d2d72f96 (127.8.254.67:37449): tablet 0b618bf0d4044e66ae16bb15c322b210 (table test-table [id=de1c6475c19f454c8ed237343f002413]) successfully deleted
I20260924 01:04:39.997946 20617 catalog_manager.cc:4841] TS de0253d4ad924b399309e23f04717c24 (127.8.254.66:34125): tablet 0b618bf0d4044e66ae16bb15c322b210 (table test-table [id=de1c6475c19f454c8ed237343f002413]) successfully deleted
I20260924 01:04:39.998654 21166 tablet_replica.cc:330] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: stopping tablet replica
I20260924 01:04:39.999190 21166 raft_consensus.cc:2229] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:04:39.999764 21166 raft_consensus.cc:2258] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:04:40.001778 21166 ts_tablet_manager.cc:1905] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 01:04:39 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:04:40.009421 21166 ts_tablet_manager.cc:1918] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:04:40.009629 21166 log.cc:1196] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/0b618bf0d4044e66ae16bb15c322b210
I20260924 01:04:40.010257 21166 ts_tablet_manager.cc:1939] T 0b618bf0d4044e66ae16bb15c322b210 P 7fb9a702ba324f618778c901239e3e1d: Deleting consensus metadata
I20260924 01:04:40.012387 20618 catalog_manager.cc:4841] TS 7fb9a702ba324f618778c901239e3e1d (127.8.254.65:39989): tablet 0b618bf0d4044e66ae16bb15c322b210 (table test-table [id=de1c6475c19f454c8ed237343f002413]) successfully deleted
I20260924 01:04:40.013831 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:39.976905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.8.254.65:39989)
0924 01:04:39.977094 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:39.977102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:39.996244 (+ 19142us) server_negotiation.cc:407] Connection header received
0924 01:04:39.996461 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:39.996467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:39.996584 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:39.996721 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:39.998603 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:39.999649 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:40.001295 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:40.001824 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:40.001977 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:40.006709 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:40.006721 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:40.006736 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:40.006783 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:40.010291 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:40.011070 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:40.011080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:40.011084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:40.011278 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:40.011697 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:40.011703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:40.011707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:40.012181 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:40.012504 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:40.013132 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 01:04:40.013400 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:04:40.014503 20839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:04:40.106411 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 20711
I20260924 01:04:40.125598 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 20850
Sep 24 01:04:40 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:04:40.142040 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:40.126460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.8.254.66:34125)
0924 01:04:40.126634 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:40.126640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:40.126767 (+ 127us) server_negotiation.cc:407] Connection header received
0924 01:04:40.126977 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:40.126993 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:40.127060 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:40.127183 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:40.128429 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:40.129377 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:40.130956 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:40.131375 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:40.131507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:40.135924 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:40.135937 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:40.135954 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:40.135992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:40.139180 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:40.139875 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:40.139883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:40.139885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:40.140027 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:40.140449 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:40.140452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:40.140454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:40.140825 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:40.141190 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:40.141441 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 01:04:40.141660 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:04:40.142814 20982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:04:40.257301 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 20850
I20260924 01:04:40.275632 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 20993
Sep 24 01:04:40 dist-test-slave-5c7l krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:04:40.292029 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:04:40.276431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.8.254.67:37449)
0924 01:04:40.276608 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 01:04:40.276612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:04:40.276631 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:04:40.276783 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:04:40.276787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:04:40.276862 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:04:40.276948 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:04:40.278213 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:40.279418 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:40.280715 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:04:40.281088 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:04:40.281180 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:04:40.286081 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:04:40.286095 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:04:40.286112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:04:40.286148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:04:40.289350 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:40.290046 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:40.290054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:40.290056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:40.290178 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:04:40.290630 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:04:40.290633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:04:40.290635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:04:40.291012 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:04:40.291213 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 01:04:40.291502 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 01:04:40.291663 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:04:40.292686 21116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:04:40.383694 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 20993
I20260924 01:04:40.400403 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 20599
I20260924 01:04:40.401623 20661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:04:40.514019 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 20599
I20260924 01:04:40.538908 9209 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 01:04:40.874509 9209 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 01:04:40.874763 9209 mini_ranger.cc:66] Stopping Ranger...
I20260924 01:04:41.220647 9209 mini_ranger.cc:68] Stopped Ranger
2026-09-24 01:04:41.221 UTC [20184] LOG: received smart shutdown request
2026-09-24 01:04:41.224 UTC [20184] LOG: background worker "logical replication launcher" (PID 20193) exited with exit code 1
2026-09-24 01:04:41.224 UTC [20188] LOG: shutting down
2026-09-24 01:04:41.407 UTC [20184] LOG: database system is shut down
2026-09-24T01:04:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54568 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:04:41 dist-test-slave-5c7l krb5kdc[21181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:04:41 dist-test-slave-5c7l krb5kdc[21181](info): set up 2 sockets
Sep 24 01:04:41 dist-test-slave-5c7l krb5kdc[21181](info): commencing operation
krb5kdc: starting...
W20260924 01:04:42.671638 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:04:42 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:04:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/rangeradmin_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/rangeradmin_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/rangerlookup_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/rangerlookup_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
I20260924 01:04:43.003324 9209 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpTDkWM/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/postgres -l logfile start
2026-09-24 01:04:45.625 UTC [21211] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 01:04:45.625 UTC [21211] LOG: listening on IPv4 address "127.8.254.84", port 43439
2026-09-24 01:04:45.629 UTC [21211] LOG: listening on Unix socket "/tmp/.s.PGSQL.43439"
2026-09-24 01:04:45.652 UTC [21214] LOG: database system was shut down at 2026-09-24 01:04:44 UTC
2026-09-24 01:04:45.656 UTC [21211] LOG: database system is ready to accept connections
I20260924 01:04:46.643324 9209 mini_postgres.cc:96] Postgres bound to 43439
2026-09-24 01:04:46.661 UTC [21222] FATAL: database "slave" does not exist
127.8.254.84:43439 - accepting connections
I20260924 01:04:46.661703 9209 mini_ranger.cc:161] Starting Ranger...
I20260924 01:04:46.688635 9209 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 01:04:47.199277 9209 mini_ranger.cc:87] Created ranger Postgres database
I20260924 01:04:47.199400 9209 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-admin
2026-09-24 01:04:47,517 [I] DB FLAVOR :POSTGRES
2026-09-24 01:04:47,517 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 01:04:47,517 [I] Checking connection..
2026-09-24 01:04:47,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 01:04:47,775 [I] Checking connection passed.
2026-09-24 01:04:47,775 [I] --------- Verifying version history table ---------
2026-09-24 01:04:47,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 01:04:48,024 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 01:04:48,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 01:04:48,287 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 01:04:48,287 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 01:04:48,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 01:04:48.550 UTC [21286] WARNING: there is no transaction in progress
2026-09-24 01:04:48,556 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 01:04:48,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 01:04:48,805 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 01:04:48,806 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 01:04:48,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 01:04:49,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 01:04:49,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5c7l.c.gcp-upstream.internal','N') ;"
2026-09-24 01:04:49,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 01:04:49,835 [I] Table x_portal_user does not exist in database ranger
2026-09-24 01:04:49,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 01:04:50,101 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 01:04:50,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 01:04:50,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 01:04:50,616 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 01:04:50,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 01:04:51.626 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.645 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.663 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.680 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.698 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.718 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.784 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.791 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.807 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.817 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.828 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.842 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.853 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.860 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.867 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:51.874 UTC [21394] WARNING: there is no transaction in progress
2026-09-24 01:04:52,537 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 01:04:52,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 01:04:52,790 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 01:04:52,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 01:04:53,060 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 01:04:53,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 01:04:53,329 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 01:04:53,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 01:04:53,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 01:04:53,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5c7l.c.gcp-upstream.internal';"
2026-09-24 01:04:54,111 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 01:04:54,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 01:04:54,377 [I] DB_PATCHES have already been applied
I20260924 01:04:54.384159 9209 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpTDkWM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 01:04:54.384250 9209 mini_ranger.cc:193] Using host: 127.8.254.84
I20260924 01:04:54.398780 9209 mini_ranger.cc:227] Ranger admin URL: http://127.8.254.84:37979
Sep 24, 2026 1:04:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:04:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:04:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:04:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:04:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:04:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:04:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 01:04:55.546185 9209 mini_ranger.cc:160] Time spent starting Ranger: real 8.884s user 0.000s sys 0.038s
Sep 24, 2026 1:05:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:05:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 1:05:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:05:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:05:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 01:05:18.005518 9209 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.254.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.254.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/rangerkms_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.254.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/rangerkms_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.254.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.254.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.254.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/HTTP_127.8.254.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 01:05:18 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211918, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 01:05:18.197407 9209 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 01:05:18.218477 9209 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 01:05:19.094651 9209 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 01:05:19.094769 9209 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms
I20260924 01:05:19.094827 9209 mini_ranger_kms.cc:227] Using postgres at 127.8.254.84:43439
2026-09-24 01:05:19,457 [I] DB FLAVOR :POSTGRES
2026-09-24 01:05:19,457 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 01:05:19,457 [I] Checking connection
2026-09-24 01:05:19,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 01:05:19,720 [I] connection success
2026-09-24 01:05:19,720 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 01:05:19,720 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 01:05:19,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 01:05:19,953 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 01:05:19,954 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 01:05:19,954 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 01:05:19,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.254.84:43439/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 01:05:20,212 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 01:05:20.719910 9209 mini_ranger_kms.cc:310] Created kms service
I20260924 01:05:20.936689 9209 mini_ranger_kms.cc:326] Created kudu user
I20260924 01:05:20.983975 9209 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 01:05:20 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211918, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:21.434754 9209 mini_ranger_kms.cc:384] Added ranger policy
I20260924 01:05:21.435083 9209 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpTDkWM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpTDkWM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 01:05:21.435150 9209 mini_ranger_kms.cc:240] Using host: 127.8.254.84
I20260924 01:05:21.450826 9209 mini_ranger_kms.cc:276] Ranger KMS PID: 21575
I20260924 01:05:21.450958 9209 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.254.84:44905
Sep 24, 2026 1:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 01:05:23 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211923, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.254.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:23.816334 9209 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.619s user 0.057s sys 0.318s
I20260924 01:05:23.816566 9209 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 01:05:23.816728 9209 mini_ranger_kms.cc:401] 127.8.254.84:44905/kms/v1/keys
Sep 24 01:05:24 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211923, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.254.84@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
Sep 24 01:05:24 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211923, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.254.84@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
Sep 24 01:05:26 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:05:26.734640 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38489
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38489
--ranger_config_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:27.064136 21625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:27.064445 21625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:27.064642 21625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:27.072016 21625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:27.072146 21625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:27.072263 21625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:27.072363 21625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:27.072453 21625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:27.083014 21625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.254.84:44905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38489
--ranger_config_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:27.084399 21625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:27.085736 21625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:27.097808 21633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:27.097817 21630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:27.097886 21631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:27.098172 21625 server_base.cc:1034] running on GCE node
I20260924 01:05:27.099030 21625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:27.100456 21625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:27.101701 21625 hybrid_clock.cc:648] HybridClock initialized: now 1790211927101668 us; error 43 us; skew 500 ppm
I20260924 01:05:27.102053 21625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:27 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211927, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:27.105108 21625 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:27.106679 21625 webserver.cc:460] Webserver started at http://127.8.254.126:43597/ using document root <none> and password file <none>
I20260924 01:05:27.107326 21625 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:27.107462 21625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:27.107775 21625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:05:27 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211927, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:27.160567 21625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "24038d17c61e44798ef0708b173507f8"
format_stamp: "Formatted at 2026-09-24 01:05:27 on dist-test-slave-5c7l"
server_key: "f0d4b3e00df1cb07603f4b5f8efac159"
server_key_iv: "15257c5a402e403fa7675631cea06dc5"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:27.161492 21625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "24038d17c61e44798ef0708b173507f8"
format_stamp: "Formatted at 2026-09-24 01:05:27 on dist-test-slave-5c7l"
server_key: "f0d4b3e00df1cb07603f4b5f8efac159"
server_key_iv: "15257c5a402e403fa7675631cea06dc5"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:27.167382 21625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:05:27.183488 21640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:27.185128 21625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:05:27.185309 21625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "24038d17c61e44798ef0708b173507f8"
format_stamp: "Formatted at 2026-09-24 01:05:27 on dist-test-slave-5c7l"
server_key: "f0d4b3e00df1cb07603f4b5f8efac159"
server_key_iv: "15257c5a402e403fa7675631cea06dc5"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:27.185529 21625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:27.198794 21625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:27.202800 21625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:27.203119 21625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:27.223254 21625 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:38489
I20260924 01:05:27.223254 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:38489 every 8 connection(s)
I20260924 01:05:27.224798 21625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:27.228827 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:27.228957 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 21625
I20260924 01:05:27.229161 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:05:27.229977 9209 external_mini_cluster.cc:1428] Setting key dafe99ca27dbe12d4a156175a4d0eb73
I20260924 01:05:27.248028 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Bootstrap starting.
I20260924 01:05:27.251914 21692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:27.253070 21692 log.cc:824] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:27.256824 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: No bootstrap required, opened a new log
Sep 24 01:05:27 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:27.267632 21692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:27.268138 21692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:27.268253 21692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24038d17c61e44798ef0708b173507f8, State: Initialized, Role: FOLLOWER
I20260924 01:05:27.268962 21692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:27.269238 21692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:05:27.269378 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:05:27.269555 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:27.232214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.8.254.126:38489)
0924 01:05:27.242827 (+ 10613us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:27.242848 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:27.243282 (+ 434us) server_negotiation.cc:407] Connection header received
0924 01:05:27.246093 (+ 2811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:27.246129 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:27.246745 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:27.247237 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:27.248895 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:27.250955 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:27.252757 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:27.253928 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:27.254138 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:27.259087 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:27.259103 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:27.259125 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:27.259177 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:27.263282 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:27.264154 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:27.264169 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:27.264178 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:27.264383 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:27.264849 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:27.264859 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:27.264862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:27.265668 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:27.265982 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:27.266497 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 01:05:27.268108 (+ 1611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10328,"thread_start_us":10080,"threads_started":1}
I20260924 01:05:27.269593 21692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:27.272379 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:27.272976 21692 leader_election.cc:302] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24038d17c61e44798ef0708b173507f8; no voters:
I20260924 01:05:27.273703 21692 leader_election.cc:288] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:05:27.273818 21697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:27.276364 21692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:27.281447 21697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 LEADER]: Becoming Leader. State: Replica: 24038d17c61e44798ef0708b173507f8, State: Running, Role: LEADER
I20260924 01:05:27.282166 21697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:27.285585 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24038d17c61e44798ef0708b173507f8. Latest consensus state: current_term: 1 leader_uuid: "24038d17c61e44798ef0708b173507f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } } }
I20260924 01:05:27.286048 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:27.286767 21698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24038d17c61e44798ef0708b173507f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } } }
I20260924 01:05:27.287099 21698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:27.299547 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:27.305627 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:05:27.306546 21692 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 01:05:27.335491 21704 catalog_manager.cc:1347] Generated new cluster ID: 9b19cb5371364ba78194c5fe6aeeab23
I20260924 01:05:27.335758 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:05:27.359068 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:05:27.360697 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:05:27.374109 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Generated new TSK 0
I20260924 01:05:27.375348 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:05:28.178756 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38489
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:05:28 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 01:05:28.585554 21738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:28.585994 21738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:28.586223 21738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:28.598127 21738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:28.598306 21738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:28.598512 21738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:05:28.614789 21738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.254.84:44905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:28.616536 21738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:28.618363 21738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:28.639178 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:28.639204 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:28.648017 21738 server_base.cc:1034] running on GCE node
W20260924 01:05:28.648434 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:28.649533 21738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:28.650825 21738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:28.652114 21738 hybrid_clock.cc:648] HybridClock initialized: now 1790211928652028 us; error 68 us; skew 500 ppm
I20260924 01:05:28.652482 21738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:28 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:28.657097 21738 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:05:28.661863 21738 webserver.cc:460] Webserver started at http://127.8.254.65:43485/ using document root <none> and password file <none>
I20260924 01:05:28.662731 21738 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:28.662890 21738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:28.663219 21738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:05:28 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:28.681077 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "7c534b76c73b45f994a42936df8f1b15"
format_stamp: "Formatted at 2026-09-24 01:05:28 on dist-test-slave-5c7l"
server_key: "3413e6ad919de4523209b8d23c207c5e"
server_key_iv: "02d89be47d4593995bf597d05cadfe04"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:28.682439 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "7c534b76c73b45f994a42936df8f1b15"
format_stamp: "Formatted at 2026-09-24 01:05:28 on dist-test-slave-5c7l"
server_key: "3413e6ad919de4523209b8d23c207c5e"
server_key_iv: "02d89be47d4593995bf597d05cadfe04"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:28.697101 21738 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.008s
I20260924 01:05:28.735474 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:28.737591 21738 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20260924 01:05:28.737902 21738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "7c534b76c73b45f994a42936df8f1b15"
format_stamp: "Formatted at 2026-09-24 01:05:28 on dist-test-slave-5c7l"
server_key: "3413e6ad919de4523209b8d23c207c5e"
server_key_iv: "02d89be47d4593995bf597d05cadfe04"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:28.738312 21738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:28.756327 21738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:28.761770 21738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:28.762092 21738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:28.763278 21738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:28.765113 21738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:28.765210 21738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:28.765388 21738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:28.765455 21738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:28.950884 21738 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:37847
I20260924 01:05:28.950914 21870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:37847 every 8 connection(s)
I20260924 01:05:28.952692 21738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:05:28.954905 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 21738
I20260924 01:05:28.955199 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:05:28.956250 9209 external_mini_cluster.cc:1428] Setting key 1e39cc87bbb7ce78182392f8160a5674
Sep 24 01:05:28 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:05:29.001487 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:28.963341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38489 (local address 127.8.254.65:42771)
0924 01:05:28.969637 (+ 6296us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:28.969700 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:28.972302 (+ 2602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:28.973515 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:28.973528 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:28.974975 (+ 1447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:28.975868 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:28.975892 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:28.977566 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:28.977571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:28.979698 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:28.979715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:28.981102 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:28.981111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:28.981515 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:28.989121 (+ 7606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:28.989162 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:28.992317 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:28.996488 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:28.996500 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:28.996523 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:28.997175 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:28.997682 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:28.997688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:28.997691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:28.997980 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:28.998748 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:28.998758 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:28.999224 (+ 466us) client_negotiation.cc:769] Sending connection context
0924 01:05:28.999528 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 01:05:29.000090 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5972,"thread_start_us":5731,"threads_started":1}
I20260924 01:05:29.002374 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:28.964028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:42771 (local address 127.8.254.126:38489)
0924 01:05:28.968988 (+ 4960us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:28.968998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:28.972748 (+ 3750us) server_negotiation.cc:407] Connection header received
0924 01:05:28.972828 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:28.972833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:28.972922 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:28.973033 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:28.976073 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:28.977403 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:28.979894 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:28.980657 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:28.980785 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:28.992690 (+ 11905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:28.992698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:28.992703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:28.992751 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:28.996281 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:28.997369 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:28.997376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:28.997380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:28.997541 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:28.998117 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:28.998122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:28.998125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:28.998518 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:28.998660 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:29.001504 (+ 2844us) server_negotiation.cc:299] Negotiation successful
0924 01:05:29.001977 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4777,"thread_start_us":4631,"threads_started":1}
I20260924 01:05:29.004133 21871 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:05:29.004491 21871 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:29.005410 21871 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:29.008020 21657 ts_manager.cc:194] Registered new tserver with Master: 7c534b76c73b45f994a42936df8f1b15 (127.8.254.65:37847)
I20260924 01:05:29.009574 21657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:42771
Sep 24 01:05:29 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:05:29.050766 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38489
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:05:29 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
W20260924 01:05:29.559346 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:29.559887 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:29.560099 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:29.572162 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:29.572424 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:29.572661 21877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:05:29.590552 21877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.254.84:44905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:29.592685 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:29.594754 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:29.633252 21885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:29.642882 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:29.649886 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:29.664613 21877 server_base.cc:1034] running on GCE node
I20260924 01:05:29.665570 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:29.668480 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:29.669772 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790211929669702 us; error 64 us; skew 500 ppm
I20260924 01:05:29.670275 21877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:29 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:29.675387 21877 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:05:29.679921 21877 webserver.cc:460] Webserver started at http://127.8.254.66:42853/ using document root <none> and password file <none>
I20260924 01:05:29.681012 21877 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:29.681188 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:29.681656 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:05:29 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:29.707466 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "70ae81563e0a4b86b16f5482040d1141"
format_stamp: "Formatted at 2026-09-24 01:05:29 on dist-test-slave-5c7l"
server_key: "0c18f72d1c8f405f2590515ae70d94be"
server_key_iv: "c9a84a6b530e9e7fb26ddab325f6110c"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:29.709252 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "70ae81563e0a4b86b16f5482040d1141"
format_stamp: "Formatted at 2026-09-24 01:05:29 on dist-test-slave-5c7l"
server_key: "0c18f72d1c8f405f2590515ae70d94be"
server_key_iv: "c9a84a6b530e9e7fb26ddab325f6110c"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:29.724222 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260924 01:05:29.748291 21901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:29.750012 21877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 01:05:29.750247 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "70ae81563e0a4b86b16f5482040d1141"
format_stamp: "Formatted at 2026-09-24 01:05:29 on dist-test-slave-5c7l"
server_key: "0c18f72d1c8f405f2590515ae70d94be"
server_key_iv: "c9a84a6b530e9e7fb26ddab325f6110c"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:29.750491 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:29.772662 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:29.777195 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:29.777539 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:29.778687 21877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:29.780259 21877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:29.780364 21877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:29.780519 21877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:29.780589 21877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:29.823858 21877 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:40461
I20260924 01:05:29.823909 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:40461 every 8 connection(s)
I20260924 01:05:29.825182 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:05:29.832436 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 21877
I20260924 01:05:29.832661 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:05:29.833114 9209 external_mini_cluster.cc:1428] Setting key 2632dd0736a56a750fba7b70cd27be94
Sep 24 01:05:29 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:29.848554 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:29.827953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:39253 (local address 127.8.254.126:38489)
0924 01:05:29.828464 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:29.828470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:29.829487 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 01:05:29.832117 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:29.832124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:29.832223 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:29.832352 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:29.834346 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:29.835442 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:29.837076 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:29.837506 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:29.837637 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:29.841560 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:29.841567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:29.841573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:29.841614 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:29.844532 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:29.845595 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:29.845604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:29.845610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:29.845762 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:29.846320 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:29.846327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:29.846334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:29.846727 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:29.846889 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:29.847779 (+ 890us) server_negotiation.cc:299] Negotiation successful
0924 01:05:29.848050 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260924 01:05:29.849568 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:29.828495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38489 (local address 127.8.254.66:39253)
0924 01:05:29.829307 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:29.829360 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:29.831750 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:29.832619 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:29.832631 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:29.833333 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:29.834140 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:29.834166 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:29.835599 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:29.835605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:29.836915 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:29.836927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:29.837750 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:29.837758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:29.837985 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:29.838735 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:29.838759 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:29.841330 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:29.844709 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:29.844719 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:29.844739 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:29.845441 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:29.845883 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:29.845888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:29.845891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:29.846170 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:29.846878 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:29.846887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:29.847325 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 01:05:29.847631 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 01:05:29.848177 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
I20260924 01:05:29.851439 22014 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38489
I20260924 01:05:29.851784 22014 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:29.852662 22014 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:29.854509 21657 ts_manager.cc:194] Registered new tserver with Master: 70ae81563e0a4b86b16f5482040d1141 (127.8.254.66:40461)
I20260924 01:05:29.855786 21657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:39253
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:05:29.894001 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38489
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:30.015326 21871 heartbeater.cc:503] Master 127.8.254.126:38489 was elected leader, sending a full tablet report...
W20260924 01:05:30.233121 22020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:30.233492 22020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:30.233683 22020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:30.241254 22020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:30.241428 22020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:30.241617 22020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:05:30.251889 22020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.254.84:44905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:30.253295 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:30.254864 22020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:30.268262 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:30.268330 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:30.268288 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:30.269640 22020 server_base.cc:1034] running on GCE node
I20260924 01:05:30.270344 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:30.271332 22020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:30.272563 22020 hybrid_clock.cc:648] HybridClock initialized: now 1790211930272516 us; error 44 us; skew 500 ppm
I20260924 01:05:30.272909 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:30 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211930, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:30.276181 22020 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:05:30.277707 22020 webserver.cc:460] Webserver started at http://127.8.254.67:39797/ using document root <none> and password file <none>
I20260924 01:05:30.278491 22020 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:30.278636 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:30.278954 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 01:05:30 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211930, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:30.293834 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "740874bbb2fb4193b71f162ed2d0f0ff"
format_stamp: "Formatted at 2026-09-24 01:05:30 on dist-test-slave-5c7l"
server_key: "e033cfed5f18756b178e006fa6e7cb1c"
server_key_iv: "11a6d230add52c1bfaca2f6530e23699"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:30.294795 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "740874bbb2fb4193b71f162ed2d0f0ff"
format_stamp: "Formatted at 2026-09-24 01:05:30 on dist-test-slave-5c7l"
server_key: "e033cfed5f18756b178e006fa6e7cb1c"
server_key_iv: "11a6d230add52c1bfaca2f6530e23699"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:30.300380 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:05:30.317718 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:30.319360 22020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 01:05:30.319545 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "740874bbb2fb4193b71f162ed2d0f0ff"
format_stamp: "Formatted at 2026-09-24 01:05:30 on dist-test-slave-5c7l"
server_key: "e033cfed5f18756b178e006fa6e7cb1c"
server_key_iv: "11a6d230add52c1bfaca2f6530e23699"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:30.319774 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:30.333935 22020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:30.337834 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:30.338153 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:30.339177 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:30.340637 22020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:30.340714 22020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:30.340891 22020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:30.340946 22020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:30.375479 22020 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:45859
I20260924 01:05:30.375519 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:45859 every 8 connection(s)
I20260924 01:05:30.376935 22020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:05:30.377614 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22020
I20260924 01:05:30.377837 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:05:30.378495 9209 external_mini_cluster.cc:1428] Setting key ca19e5c775325f413da42a458ccde136
Sep 24 01:05:30 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211930, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:30.397850 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:30.380046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:49353 (local address 127.8.254.126:38489)
0924 01:05:30.380530 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:30.380535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:30.381751 (+ 1216us) server_negotiation.cc:407] Connection header received
0924 01:05:30.383703 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:30.383709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:30.383806 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:30.383935 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:30.385347 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.386057 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.387671 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.388126 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.388273 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:30.391716 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:30.391720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:30.391724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:30.391756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:30.395048 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.395846 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.395850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.395852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.395973 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.396367 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.396370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.396372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.396624 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:30.396741 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:30.397435 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 01:05:30.397600 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260924 01:05:30.398891 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:30.380772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38489 (local address 127.8.254.67:49353)
0924 01:05:30.381581 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:30.381637 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:30.383442 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:30.384170 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:30.384179 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:30.384598 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:30.385178 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:30.385197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.386250 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.386256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:30.387511 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:30.387524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.388364 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.388371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:30.388607 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:30.389292 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:30.389314 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:30.391505 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:30.395197 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:30.395204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:30.395218 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:30.395728 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:30.396066 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:30.396071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:30.396073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:30.396268 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:30.396766 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:30.396781 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:30.397098 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 01:05:30.397336 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 01:05:30.397726 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260924 01:05:30.400189 22148 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38489
I20260924 01:05:30.400501 22148 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:30.401473 22148 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:30.402927 21657 ts_manager.cc:194] Registered new tserver with Master: 740874bbb2fb4193b71f162ed2d0f0ff (127.8.254.67:45859)
I20260924 01:05:30.403733 21657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:49353
I20260924 01:05:30.407461 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:05:30.409124 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 21738
Sep 24 01:05:30 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:05:30.426877 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:30.410150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.8.254.65:37847)
0924 01:05:30.410829 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:30.410835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:30.410860 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:05:30.410918 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:30.410925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:30.411136 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:30.411259 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:30.412875 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.413958 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.415484 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.416027 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.416131 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:30.420904 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:30.420917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:30.420931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:30.420965 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:30.423844 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.424573 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.424580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.424582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.424736 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.425154 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.425159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.425161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.425505 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:30.425833 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:30.426153 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 01:05:30.426567 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":346,"threads_started":1}
I20260924 01:05:30.428180 21866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:30.511937 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 21738
I20260924 01:05:30.528831 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 21877
Sep 24 01:05:30 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:05:30.550720 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:30.529708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.8.254.66:40461)
0924 01:05:30.530327 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:30.530334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:30.530360 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:05:30.530422 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:30.530428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:30.530704 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:30.530878 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:30.532112 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.533250 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.534900 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.535448 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.535608 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:30.544419 (+ 8811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:30.544436 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:30.544452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:30.544498 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:30.547581 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.548422 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.548431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.548433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.548583 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.548993 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.548999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.549003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.549378 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:30.549703 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:30.550043 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 01:05:30.550349 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
I20260924 01:05:30.552069 22009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:30.635385 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 21877
I20260924 01:05:30.651376 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22020
Sep 24 01:05:30 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:05:30.669340 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:30.652399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.8.254.67:45859)
0924 01:05:30.653089 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:30.653095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:30.653125 (+ 30us) server_negotiation.cc:407] Connection header received
0924 01:05:30.653189 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:30.653195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:30.653427 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:30.653624 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:30.655002 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.656037 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.657994 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:30.658717 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:30.658868 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:30.663330 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:30.663343 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:30.663359 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:30.663403 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:30.666243 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.667043 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.667057 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.667060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.667201 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:30.667607 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:30.667613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:30.667616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:30.667986 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:30.668342 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:30.668673 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 01:05:30.668973 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":388,"threads_started":1}
I20260924 01:05:30.670877 22143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:30.755951 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22020
I20260924 01:05:30.772132 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 21625
I20260924 01:05:30.773459 21687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:30.880370 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 21625
I20260924 01:05:30.905249 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38489
--webserver_interface=127.8.254.126
--webserver_port=43597
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:38489
--ranger_config_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:31.233465 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:31.233803 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:31.233891 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:31.241442 22160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:31.241561 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:31.241618 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:31.241668 22160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:31.241719 22160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:31.252913 22160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:38489
--ranger_config_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=43597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:31.254315 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:31.255638 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:31.268085 22166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:31.268146 22165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:31.268184 22168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:31.268786 22160 server_base.cc:1034] running on GCE node
I20260924 01:05:31.269776 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:31.271409 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:31.272625 22160 hybrid_clock.cc:648] HybridClock initialized: now 1790211931272558 us; error 55 us; skew 500 ppm
I20260924 01:05:31.273030 22160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:31 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:31.276346 22160 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:31.278028 22160 webserver.cc:460] Webserver started at http://127.8.254.126:43597/ using document root <none> and password file <none>
I20260924 01:05:31.278824 22160 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:31.278980 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 01:05:31 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:31.337566 22160 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 01:05:31.340909 22176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:31.342545 22160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 01:05:31.342774 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "24038d17c61e44798ef0708b173507f8"
format_stamp: "Formatted at 2026-09-24 01:05:27 on dist-test-slave-5c7l"
server_key: "f0d4b3e00df1cb07603f4b5f8efac159"
server_key_iv: "15257c5a402e403fa7675631cea06dc5"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:31.343822 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:31.365054 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:31.369453 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:31.369782 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:31.388912 22160 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:38489
I20260924 01:05:31.388933 22227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:38489 every 8 connection(s)
I20260924 01:05:31.390489 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:31.397290 22228 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:05:31.398774 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22160
I20260924 01:05:31.399629 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:37847
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:38489
--webserver_port=43485
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:31.400842 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Bootstrap starting.
I20260924 01:05:31.426119 22228 log.cc:824] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:31.437842 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:05:31.438571 22228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Bootstrap complete.
I20260924 01:05:31.451777 22228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:31.453047 22228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24038d17c61e44798ef0708b173507f8, State: Initialized, Role: FOLLOWER
I20260924 01:05:31.453858 22228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:31.454144 22228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:05:31.454293 22228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:05:31.454532 22228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 1 FOLLOWER]: Advancing to term 2
I20260924 01:05:31.460562 22228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:31.461267 22228 leader_election.cc:302] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24038d17c61e44798ef0708b173507f8; no voters:
I20260924 01:05:31.462018 22228 leader_election.cc:288] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 01:05:31.462177 22232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 01:05:31.463551 22232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [term 2 LEADER]: Becoming Leader. State: Replica: 24038d17c61e44798ef0708b173507f8, State: Running, Role: LEADER
I20260924 01:05:31.464262 22228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:31.464179 22232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } }
I20260924 01:05:31.466675 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24038d17c61e44798ef0708b173507f8. Latest consensus state: current_term: 2 leader_uuid: "24038d17c61e44798ef0708b173507f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } } }
I20260924 01:05:31.467067 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:31.467602 22233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "24038d17c61e44798ef0708b173507f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24038d17c61e44798ef0708b173507f8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 38489 } } }
I20260924 01:05:31.467898 22233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:31.469710 22238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:31.475528 22238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:05:31.476994 22238 catalog_manager.cc:1259] Loaded cluster ID: 9b19cb5371364ba78194c5fe6aeeab23
I20260924 01:05:31.477149 22238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:05:31.481261 22238 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:05:31.482765 22238 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Loaded TSK: 0
I20260924 01:05:31.484376 22238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:05:31.494338 22228 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 01:05:31.846882 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:31.847442 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:31.847676 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:31.859761 22230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:31.860008 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:31.860266 22230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:05:31.876407 22230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:37847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=43485
--tserver_master_addrs=127.8.254.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:31.878288 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:31.880244 22230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:31.916352 22265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:31.916445 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:31.926764 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:31.931217 22230 server_base.cc:1034] running on GCE node
I20260924 01:05:31.932327 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:31.937791 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:31.939180 22230 hybrid_clock.cc:648] HybridClock initialized: now 1790211931939094 us; error 82 us; skew 500 ppm
I20260924 01:05:31.939800 22230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:31 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:31.944478 22230 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:05:31.951969 22230 webserver.cc:460] Webserver started at http://127.8.254.65:43485/ using document root <none> and password file <none>
I20260924 01:05:31.953163 22230 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:31.953423 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 01:05:31 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:32.000605 22230 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.010s sys 0.000s
I20260924 01:05:32.011281 22276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:32.013545 22230 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.003s
I20260924 01:05:32.013870 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "7c534b76c73b45f994a42936df8f1b15"
format_stamp: "Formatted at 2026-09-24 01:05:28 on dist-test-slave-5c7l"
server_key: "3413e6ad919de4523209b8d23c207c5e"
server_key_iv: "02d89be47d4593995bf597d05cadfe04"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:32.015568 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:32.084080 22230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:32.092876 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:32.093300 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:32.106705 22230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:32.125129 22230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:32.125360 22230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 01:05:32.125582 22230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:32.125746 22230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:32.284394 22230 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:37847
I20260924 01:05:32.284466 22389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:37847 every 8 connection(s)
I20260924 01:05:32.286536 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:05:32.293061 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22230
I20260924 01:05:32.293954 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:40461
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:38489
--webserver_port=42853
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:05:32 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:32.340871 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:32.296886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38489 (local address 127.8.254.65:56263)
0924 01:05:32.306870 (+ 9984us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:32.306929 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:32.310080 (+ 3151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:32.314009 (+ 3929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:32.314026 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:32.314971 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:32.316003 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:32.316038 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:32.322694 (+ 6656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:32.322701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:32.324939 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:32.324969 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:32.327057 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:32.327064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:32.327335 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:32.328155 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:32.328185 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:32.330751 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:32.335262 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:32.335272 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:32.335290 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:32.335869 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:32.336560 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:32.336565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:32.336569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:32.336828 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:32.337944 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:32.337952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:32.338612 (+ 660us) client_negotiation.cc:769] Sending connection context
0924 01:05:32.338912 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 01:05:32.339481 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9648,"thread_start_us":9400,"threads_started":1}
I20260924 01:05:32.343609 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:32.298062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:56263 (local address 127.8.254.126:38489)
0924 01:05:32.311119 (+ 13057us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:32.311146 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:32.311214 (+ 68us) server_negotiation.cc:407] Connection header received
0924 01:05:32.311825 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:32.311850 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:32.312521 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:32.313053 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:32.316366 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:32.318942 (+ 2576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:32.325151 (+ 6209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:32.326433 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:32.326645 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:32.331116 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:32.331128 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:32.331152 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:32.331203 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:32.335061 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:32.336045 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:32.336055 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:32.336063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:32.336409 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:32.336965 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:32.336970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:32.336973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:32.337673 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:32.337852 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:32.341813 (+ 3961us) server_negotiation.cc:299] Negotiation successful
0924 01:05:32.342321 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12753,"thread_start_us":12507,"threads_started":1}
I20260924 01:05:32.349866 22390 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38489
I20260924 01:05:32.350375 22390 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:32.351406 22390 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
W20260924 01:05:32.355460 22390 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:38489 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 01:05:32.355708 22390 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:32.356021 22390 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:32.357471 22390 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:32.357833 22390 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
W20260924 01:05:32.359262 22390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:05:32.732069 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:32.732404 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:32.732584 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:32.740331 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:32.740490 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:32.740702 22394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:05:32.752832 22394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:40461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=42853
--tserver_master_addrs=127.8.254.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:32.754820 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:32.756551 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:32.805925 22410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:32.807524 22413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:32.807765 22408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:32.808612 22394 server_base.cc:1034] running on GCE node
I20260924 01:05:32.809597 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:32.818748 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:32.820453 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790211932820370 us; error 82 us; skew 500 ppm
I20260924 01:05:32.820950 22394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:32 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:32.830633 22394 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:05:32.832712 22394 webserver.cc:460] Webserver started at http://127.8.254.66:42853/ using document root <none> and password file <none>
I20260924 01:05:32.833635 22394 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:32.833798 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 01:05:32 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:32.873625 22394 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.005s sys 0.000s
I20260924 01:05:32.890864 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:32.895452 22394 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.013s sys 0.003s
I20260924 01:05:32.895769 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "70ae81563e0a4b86b16f5482040d1141"
format_stamp: "Formatted at 2026-09-24 01:05:29 on dist-test-slave-5c7l"
server_key: "0c18f72d1c8f405f2590515ae70d94be"
server_key_iv: "c9a84a6b530e9e7fb26ddab325f6110c"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:32.897431 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 24 01:05:32 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:32.977779 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:32.982869 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:32.983150 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:32.984288 22394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:32.985842 22394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:32.985970 22394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:32.986099 22394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:32.986184 22394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:33.042809 22394 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:40461
I20260924 01:05:33.042855 22534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:40461 every 8 connection(s)
I20260924 01:05:33.044394 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:05:33.050894 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22394
I20260924 01:05:33.051698 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.254.84:44905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:45859
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:38489
--webserver_port=39797
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:05:33 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:33.073040 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:33.047821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:52371 (local address 127.8.254.126:38489)
0924 01:05:33.048401 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:33.048408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:33.049430 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 01:05:33.052062 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:33.052071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:33.052173 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:33.052258 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:33.054778 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:33.056012 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:33.058677 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:33.059456 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:33.059585 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:33.064915 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:33.064923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:33.064929 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:33.064972 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:33.068772 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:33.069867 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:33.069875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:33.069878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:33.070037 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:33.070685 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:33.070690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:33.070695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:33.071118 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:33.071300 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:33.072352 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0924 01:05:33.072612 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
I20260924 01:05:33.074496 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:33.048347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38489 (local address 127.8.254.66:52371)
0924 01:05:33.049266 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:33.049313 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:33.051673 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:33.052899 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:33.052914 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:33.053606 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:33.054535 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:33.054566 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:33.056247 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:33.056254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:33.058489 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:33.058516 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:33.059984 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:33.059989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:33.060196 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:33.062064 (+ 1868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:33.062093 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:33.064576 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:33.068957 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:33.068969 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:33.068993 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:33.069683 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:33.070190 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:33.070219 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:33.070223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:33.070520 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:33.071315 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:33.071326 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:33.071856 (+ 530us) client_negotiation.cc:769] Sending connection context
0924 01:05:33.072266 (+ 410us) client_negotiation.cc:240] Negotiation successful
0924 01:05:33.072896 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":420,"threads_started":1}
I20260924 01:05:33.077085 22535 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38489
I20260924 01:05:33.077466 22535 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:33.078500 22535 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:33.080977 22193 ts_manager.cc:194] Registered new tserver with Master: 70ae81563e0a4b86b16f5482040d1141 (127.8.254.66:40461)
I20260924 01:05:33.083559 22193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:52371
I20260924 01:05:33.359439 22390 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:33.359910 22390 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:33.361475 22193 ts_manager.cc:194] Registered new tserver with Master: 7c534b76c73b45f994a42936df8f1b15 (127.8.254.65:37847)
I20260924 01:05:33.362838 22193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:56263
Sep 24 01:05:33 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
W20260924 01:05:33.492141 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:33.492619 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:33.492794 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:33.500712 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:33.500867 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:33.501042 22539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:05:33.514784 22539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.254.84:44905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:45859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=39797
--tserver_master_addrs=127.8.254.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:33.516419 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:33.518049 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:33.533990 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:33.538980 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:33.542809 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:33.543444 22539 server_base.cc:1034] running on GCE node
I20260924 01:05:33.544195 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:33.545409 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:33.546696 22539 hybrid_clock.cc:648] HybridClock initialized: now 1790211933546624 us; error 60 us; skew 500 ppm
I20260924 01:05:33.547082 22539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:33 dist-test-slave-5c7l krb5kdc[21181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211933, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:33.551188 22539 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:05:33.555879 22539 webserver.cc:460] Webserver started at http://127.8.254.67:39797/ using document root <none> and password file <none>
I20260924 01:05:33.556833 22539 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:33.556983 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 01:05:33 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211933, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:33.582660 22539 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:05:33.586416 22555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:33.588249 22539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 01:05:33.588495 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "740874bbb2fb4193b71f162ed2d0f0ff"
format_stamp: "Formatted at 2026-09-24 01:05:30 on dist-test-slave-5c7l"
server_key: "e033cfed5f18756b178e006fa6e7cb1c"
server_key_iv: "11a6d230add52c1bfaca2f6530e23699"
server_key_version: "kuduclusterkey@0"
I20260924 01:05:33.589982 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:33.647933 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:33.653712 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:33.654138 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:33.665418 22539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:33.673266 22539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:33.673501 22539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 01:05:33.673722 22539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:33.673885 22539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 01:05:33 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for HTTP/127.8.254.84@KRBTEST.COM
I20260924 01:05:33.923388 22539 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:45859
I20260924 01:05:33.923470 22668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:45859 every 8 connection(s)
I20260924 01:05:33.925493 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:05:33.932602 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22539
Sep 24 01:05:34 dist-test-slave-5c7l krb5kdc[21181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211933, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:34.021549 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:33.934244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.8.254.126:38489)
0924 01:05:33.990216 (+ 55972us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:33.990223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:33.990244 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:05:33.990323 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:33.990330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:33.990436 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:33.990567 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:34.000954 (+ 10387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.003469 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.005741 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.006699 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.006882 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.012364 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:34.012372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:34.012377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:34.012428 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:34.017840 (+ 5412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.018948 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.018953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.018955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.019085 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.019568 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.019574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.019578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.019875 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:34.020246 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.020636 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.020913 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55837,"thread_start_us":53743,"threads_started":1}
I20260924 01:05:34.027984 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:33.929767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:36257 (local address 127.8.254.126:38489)
0924 01:05:33.987968 (+ 58201us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:33.987975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.000841 (+ 12866us) server_negotiation.cc:407] Connection header received
0924 01:05:34.007370 (+ 6529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.007379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.007500 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.007629 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:34.010497 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.011723 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.015340 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.016011 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.016154 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.020251 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:34.020259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:34.020264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:34.020305 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:34.024142 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.025238 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.025244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.025248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.025393 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.025916 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.025921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.025925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.026287 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:34.026429 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.027347 (+ 918us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.027583 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58057,"thread_start_us":60903,"threads_started":1}
I20260924 01:05:34.028947 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:33.929137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:38489 (local address 127.8.254.67:36257)
0924 01:05:33.998937 (+ 69800us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:33.999003 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:34.007003 (+ 8000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:34.008331 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:34.008351 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:34.009124 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:34.010219 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.010251 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.012078 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.012098 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.015128 (+ 3030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.015187 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.016295 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.016301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.016544 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.017298 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:34.017329 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:34.019930 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:34.024309 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:34.024319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:34.024474 (+ 155us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:34.025095 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:34.025503 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:34.025507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:34.025509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:34.025790 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:34.026506 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:34.026514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:34.026948 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 01:05:34.027213 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 01:05:34.027675 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69400,"thread_start_us":73084,"threads_started":1}
I20260924 01:05:34.030782 22669 heartbeater.cc:347] Connected to a master server at 127.8.254.126:38489
I20260924 01:05:34.031126 22669 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:34.031924 22669 heartbeater.cc:511] Master 127.8.254.126:38489 requested a full tablet report, sending...
I20260924 01:05:34.033555 22193 ts_manager.cc:194] Registered new tserver with Master: 740874bbb2fb4193b71f162ed2d0f0ff (127.8.254.67:45859)
I20260924 01:05:34.034554 22193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:36257
I20260924 01:05:34.037360 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:05:34.057039 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.040912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.8.254.126:38489)
0924 01:05:34.041133 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.041141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.042140 (+ 999us) server_negotiation.cc:407] Connection header received
0924 01:05:34.042244 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.042249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.042327 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.042431 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:34.044398 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.045488 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.047531 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.048022 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.048124 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.051557 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:34.051564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:34.051567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:34.051603 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:34.054325 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.055127 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.055131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.055133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.055258 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.055701 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.055704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.055706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.055973 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:34.056144 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.056618 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.056795 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:05:34.062753 22193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:05:34.070662 22193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:05:34.088616 22535 heartbeater.cc:503] Master 127.8.254.126:38489 was elected leader, sending a full tablet report...
I20260924 01:05:34.097327 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.085587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56730)
0924 01:05:34.086457 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.086498 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:34.086694 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:34.088313 (+ 1619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:34.088321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:34.088355 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:34.088757 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.088770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.091101 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.091107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.093883 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.093906 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.096267 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.096276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.096505 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.096564 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 01:05:34.096680 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 01:05:34.096896 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":819,"threads_started":1}
I20260924 01:05:34.099069 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.086776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.8.254.65:37847)
0924 01:05:34.087622 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.087632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.087661 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:05:34.087765 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.087773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.088035 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.088981 (+ 946us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:34.089226 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.090935 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.094166 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.095973 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.098178 (+ 2205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.098468 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.098543 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.098718 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":774,"threads_started":1}
I20260924 01:05:34.103802 22325 tablet_service.cc:1467] Processing CreateTablet for tablet 5c266185732f43b3a5d9ad7543d628c9 (DEFAULT_TABLE table=test-table [id=5ff99ebd08424fb283b6e0115ea6ced7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:34.105904 22325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c266185732f43b3a5d9ad7543d628c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:34.116565 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.087411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:40461 (local address 127.0.0.1:57716)
0924 01:05:34.101413 (+ 14002us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.101436 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:34.101572 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:34.103452 (+ 1880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:34.103457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:34.103485 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:34.103875 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.103889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.107757 (+ 3868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.107766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.110710 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.110731 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.115427 (+ 4696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.115434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.115855 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.115889 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:05:34.115973 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 01:05:34.116130 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13849,"mutex_wait_us":75,"thread_start_us":14254,"threads_started":1}
I20260924 01:05:34.118785 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.086245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:45859 (local address 127.0.0.1:33114)
0924 01:05:34.097868 (+ 11623us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.097890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:34.098004 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:34.106145 (+ 8141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:34.106151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:34.106179 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:34.106397 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.106408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.114360 (+ 7952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.114366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.116155 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.116169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.118012 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.118020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.118223 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.118267 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 01:05:34.118340 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 01:05:34.118466 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11510,"mutex_wait_us":23,"thread_start_us":27121,"threads_started":1}
I20260924 01:05:34.119024 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.087881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.8.254.66:40461)
0924 01:05:34.102539 (+ 14658us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.102550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.102578 (+ 28us) server_negotiation.cc:407] Connection header received
0924 01:05:34.102677 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.102685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.102955 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.103171 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:34.105449 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.107526 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.113646 (+ 6120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.114890 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.115026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.115253 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.117171 (+ 1918us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.117365 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14502,"thread_start_us":14337,"threads_started":1}
I20260924 01:05:34.120334 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.087337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.8.254.67:45859)
0924 01:05:34.104408 (+ 17071us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.104417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.104443 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:05:34.104560 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.104571 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.104822 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.105047 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:34.110980 (+ 5933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.113050 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.116342 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.117809 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.119492 (+ 1683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.119716 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.119789 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.119950 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16909,"thread_start_us":16788,"threads_started":1}
I20260924 01:05:34.123652 22470 tablet_service.cc:1467] Processing CreateTablet for tablet 5c266185732f43b3a5d9ad7543d628c9 (DEFAULT_TABLE table=test-table [id=5ff99ebd08424fb283b6e0115ea6ced7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:34.125314 22470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c266185732f43b3a5d9ad7543d628c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:34.124912 22604 tablet_service.cc:1467] Processing CreateTablet for tablet 5c266185732f43b3a5d9ad7543d628c9 (DEFAULT_TABLE table=test-table [id=5ff99ebd08424fb283b6e0115ea6ced7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:34.126366 22604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c266185732f43b3a5d9ad7543d628c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:34.135027 22697 tablet_bootstrap.cc:492] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Bootstrap starting.
I20260924 01:05:34.139150 22697 tablet_bootstrap.cc:654] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:34.140720 22697 log.cc:824] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:34.146466 22697 tablet_bootstrap.cc:492] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: No bootstrap required, opened a new log
I20260924 01:05:34.146839 22697 ts_tablet_manager.cc:1397] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 01:05:34.150988 22699 tablet_bootstrap.cc:492] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Bootstrap starting.
I20260924 01:05:34.155325 22699 tablet_bootstrap.cc:654] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:34.156977 22699 log.cc:824] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:34.160048 22697 raft_consensus.cc:348] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.160915 22697 raft_consensus.cc:374] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:34.161108 22697 raft_consensus.cc:729] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c534b76c73b45f994a42936df8f1b15, State: Initialized, Role: FOLLOWER
I20260924 01:05:34.162307 22697 consensus_queue.cc:260] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.163106 22700 tablet_bootstrap.cc:492] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Bootstrap starting.
I20260924 01:05:34.167111 22700 tablet_bootstrap.cc:654] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:34.167735 22699 tablet_bootstrap.cc:492] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: No bootstrap required, opened a new log
I20260924 01:05:34.168195 22699 ts_tablet_manager.cc:1397] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260924 01:05:34.168437 22700 log.cc:824] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:34.171285 22697 ts_tablet_manager.cc:1428] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Time spent starting tablet: real 0.024s user 0.014s sys 0.009s
I20260924 01:05:34.171748 22390 heartbeater.cc:503] Master 127.8.254.126:38489 was elected leader, sending a full tablet report...
I20260924 01:05:34.176610 22700 tablet_bootstrap.cc:492] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: No bootstrap required, opened a new log
I20260924 01:05:34.177011 22700 ts_tablet_manager.cc:1397] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260924 01:05:34.179745 22699 raft_consensus.cc:348] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.180184 22699 raft_consensus.cc:374] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:34.180311 22699 raft_consensus.cc:729] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ae81563e0a4b86b16f5482040d1141, State: Initialized, Role: FOLLOWER
I20260924 01:05:34.180931 22699 consensus_queue.cc:260] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.182761 22699 ts_tablet_manager.cc:1428] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260924 01:05:34.187942 22700 raft_consensus.cc:348] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.188354 22700 raft_consensus.cc:374] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:34.188463 22700 raft_consensus.cc:729] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740874bbb2fb4193b71f162ed2d0f0ff, State: Initialized, Role: FOLLOWER
I20260924 01:05:34.189119 22700 consensus_queue.cc:260] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.190369 22669 heartbeater.cc:503] Master 127.8.254.126:38489 was elected leader, sending a full tablet report...
I20260924 01:05:34.190955 22700 ts_tablet_manager.cc:1428] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 01:05:34.210574 22705 raft_consensus.cc:482] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:05:34.210898 22705 raft_consensus.cc:504] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.214859 22705 leader_election.cc:288] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c534b76c73b45f994a42936df8f1b15 (127.8.254.65:37847), 70ae81563e0a4b86b16f5482040d1141 (127.8.254.66:40461)
I20260924 01:05:34.221060 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.214111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:36569 (local address 127.8.254.66:40461)
0924 01:05:34.214330 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.214337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.215001 (+ 664us) server_negotiation.cc:407] Connection header received
0924 01:05:34.215254 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.215259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.215351 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.215447 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:34.216049 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.217009 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.219199 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.219986 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.220176 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.220251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.220612 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.220775 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 01:05:34.221061 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.213346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.8.254.67:57869)
0924 01:05:34.213780 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.213801 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:34.213939 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:34.214990 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:34.214994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:34.215021 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:34.215260 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.215269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.216551 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.216556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.218518 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.218533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.220351 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.220356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.220557 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.220584 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 01:05:34.220653 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 01:05:34.220757 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":86}
I20260924 01:05:34.221115 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.213516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:40461 (local address 127.8.254.67:36569)
0924 01:05:34.214902 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.214921 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:34.215099 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:34.215490 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:34.215494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:34.215521 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:34.215884 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.215895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.217173 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.217179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.219049 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:34.219060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.220178 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.220184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:34.220418 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.220455 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 01:05:34.220551 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 01:05:34.220694 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"mutex_wait_us":38,"thread_start_us":1032,"threads_started":1}
I20260924 01:05:34.221884 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.214290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:57869 (local address 127.8.254.65:37847)
0924 01:05:34.214502 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.214508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.214530 (+ 22us) server_negotiation.cc:407] Connection header received
0924 01:05:34.214612 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.214620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.214697 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.214795 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:34.215407 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.216406 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.218747 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.220016 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.220199 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.220287 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.221396 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.221583 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:05:34.222709 22490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c266185732f43b3a5d9ad7543d628c9" candidate_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ae81563e0a4b86b16f5482040d1141" is_pre_election: true
I20260924 01:05:34.223007 22345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c266185732f43b3a5d9ad7543d628c9" candidate_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c534b76c73b45f994a42936df8f1b15" is_pre_election: true
I20260924 01:05:34.223301 22490 raft_consensus.cc:2454] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 740874bbb2fb4193b71f162ed2d0f0ff in term 0.
I20260924 01:05:34.223549 22345 raft_consensus.cc:2454] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 740874bbb2fb4193b71f162ed2d0f0ff in term 0.
I20260924 01:05:34.224061 22557 leader_election.cc:302] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ae81563e0a4b86b16f5482040d1141, 740874bbb2fb4193b71f162ed2d0f0ff; no voters:
I20260924 01:05:34.224470 22705 raft_consensus.cc:2781] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:05:34.224623 22705 raft_consensus.cc:482] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:05:34.224720 22705 raft_consensus.cc:3037] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:34.227836 22705 raft_consensus.cc:504] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.228538 22705 leader_election.cc:288] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [CANDIDATE]: Term 1 election: Requested vote from peers 7c534b76c73b45f994a42936df8f1b15 (127.8.254.65:37847), 70ae81563e0a4b86b16f5482040d1141 (127.8.254.66:40461)
I20260924 01:05:34.229049 22490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c266185732f43b3a5d9ad7543d628c9" candidate_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ae81563e0a4b86b16f5482040d1141"
I20260924 01:05:34.229096 22345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c266185732f43b3a5d9ad7543d628c9" candidate_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c534b76c73b45f994a42936df8f1b15"
I20260924 01:05:34.229275 22490 raft_consensus.cc:3037] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:34.229362 22345 raft_consensus.cc:3037] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:34.231674 22490 raft_consensus.cc:2454] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 740874bbb2fb4193b71f162ed2d0f0ff in term 1.
I20260924 01:05:34.232161 22557 leader_election.cc:302] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ae81563e0a4b86b16f5482040d1141, 740874bbb2fb4193b71f162ed2d0f0ff; no voters:
I20260924 01:05:34.232463 22705 raft_consensus.cc:2781] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:34.232514 22345 raft_consensus.cc:2454] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 740874bbb2fb4193b71f162ed2d0f0ff in term 1.
I20260924 01:05:34.233179 22705 raft_consensus.cc:686] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 1 LEADER]: Becoming Leader. State: Replica: 740874bbb2fb4193b71f162ed2d0f0ff, State: Running, Role: LEADER
I20260924 01:05:34.233712 22705 consensus_queue.cc:237] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } }
I20260924 01:05:34.239033 22193 catalog_manager.cc:5495] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 740874bbb2fb4193b71f162ed2d0f0ff (127.8.254.67). New cstate: current_term: 1 leader_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "740874bbb2fb4193b71f162ed2d0f0ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 } health_report { overall_health: UNKNOWN } } }
W20260924 01:05:34.259577 22670 tablet.cc:2369] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:34.282650 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.276044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.8.254.67:45859)
0924 01:05:34.276263 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.276268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.276333 (+ 65us) server_negotiation.cc:407] Connection header received
0924 01:05:34.276552 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.276556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.276635 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.276766 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:05:34.277176 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.277897 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.279923 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.280568 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.280692 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.281340 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:05:34.282013 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:05:34.282173 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.282291 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.282392 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 01:05:34.293131 22345 raft_consensus.cc:1261] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 1 FOLLOWER]: Refusing update from remote peer 740874bbb2fb4193b71f162ed2d0f0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:34.293131 22490 raft_consensus.cc:1261] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 1 FOLLOWER]: Refusing update from remote peer 740874bbb2fb4193b71f162ed2d0f0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:34.294301 22707 consensus_queue.cc:1035] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ae81563e0a4b86b16f5482040d1141" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:34.294749 22705 consensus_queue.cc:1035] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c534b76c73b45f994a42936df8f1b15" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 01:05:34.298586 22536 tablet.cc:2369] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:34.310842 22710 mvcc.cc:205] Tried to move back new op lower bound from 7332708082853228544 to 7332708082625945600. Current Snapshot: MvccSnapshot[applied={T|T < 7332708082853228544}]
I20260924 01:05:34.311260 22711 mvcc.cc:205] Tried to move back new op lower bound from 7332708082853228544 to 7332708082625945600. Current Snapshot: MvccSnapshot[applied={T|T < 7332708082853228544}]
W20260924 01:05:34.324493 22393 tablet.cc:2369] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:34.340533 22192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:05:34.340886 22192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:05:34.345361 22192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24038d17c61e44798ef0708b173507f8: Sending DeleteTablet for 3 replicas of tablet 5c266185732f43b3a5d9ad7543d628c9
I20260924 01:05:34.346704 22325 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c266185732f43b3a5d9ad7543d628c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:34 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:56730
I20260924 01:05:34.346832 22470 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c266185732f43b3a5d9ad7543d628c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:34 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:57716
I20260924 01:05:34.347976 22604 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c266185732f43b3a5d9ad7543d628c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:34 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:33114
I20260924 01:05:34.356842 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22230
I20260924 01:05:34.367303 22720 tablet_replica.cc:330] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: stopping tablet replica
I20260924 01:05:34.367923 22722 tablet_replica.cc:330] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: stopping tablet replica
I20260924 01:05:34.368167 22720 raft_consensus.cc:2229] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:05:34.368494 22722 raft_consensus.cc:2229] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:05:34.368911 22720 raft_consensus.cc:2258] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:34.369306 22722 raft_consensus.cc:2258] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:34.372780 22720 ts_tablet_manager.cc:1905] T 5c266185732f43b3a5d9ad7543d628c9 P 7c534b76c73b45f994a42936df8f1b15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:34.376295 22722 ts_tablet_manager.cc:1905] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:34.376557 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.358615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.8.254.65:37847)
0924 01:05:34.358823 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.358831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.358852 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:05:34.358924 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.358932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.359019 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.359132 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:34.361150 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.362322 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.364712 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.365393 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.365535 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.369170 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:34.369183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:34.369203 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:34.369249 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:34.373163 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.374119 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.374129 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.374133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.374347 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.374761 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.374767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.374772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.375270 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:34.375582 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.375893 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.376150 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 01:05:34.375664 22721 tablet_replica.cc:330] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: stopping tablet replica
I20260924 01:05:34.377251 22385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:34.377873 22721 raft_consensus.cc:2229] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:05:34.378473 22721 raft_consensus.cc:2258] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:34.380541 22721 ts_tablet_manager.cc:1905] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:34.383961 22722 ts_tablet_manager.cc:1918] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:34.387531 22721 ts_tablet_manager.cc:1918] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:34.387754 22721 log.cc:1196] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/5c266185732f43b3a5d9ad7543d628c9
I20260924 01:05:34.388317 22721 ts_tablet_manager.cc:1939] T 5c266185732f43b3a5d9ad7543d628c9 P 70ae81563e0a4b86b16f5482040d1141: Deleting consensus metadata
I20260924 01:05:34.390269 22178 catalog_manager.cc:4841] TS 70ae81563e0a4b86b16f5482040d1141 (127.8.254.66:40461): tablet 5c266185732f43b3a5d9ad7543d628c9 (table test-table [id=5ff99ebd08424fb283b6e0115ea6ced7]) successfully deleted
I20260924 01:05:34.394336 22722 log.cc:1196] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/5c266185732f43b3a5d9ad7543d628c9
I20260924 01:05:34.395030 22722 ts_tablet_manager.cc:1939] T 5c266185732f43b3a5d9ad7543d628c9 P 740874bbb2fb4193b71f162ed2d0f0ff: Deleting consensus metadata
I20260924 01:05:34.396942 22180 catalog_manager.cc:4841] TS 740874bbb2fb4193b71f162ed2d0f0ff (127.8.254.67:45859): tablet 5c266185732f43b3a5d9ad7543d628c9 (table test-table [id=5ff99ebd08424fb283b6e0115ea6ced7]) successfully deleted
I20260924 01:05:34.518967 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22230
W20260924 01:05:34.536927 22177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.254.65:37847 (error 108)
I20260924 01:05:34.537688 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22394
I20260924 01:05:34.538914 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.538299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56750)
0924 01:05:34.538583 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.538709 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260924 01:05:34.539165 22177 catalog_manager.cc:4584] TS 7c534b76c73b45f994a42936df8f1b15 (127.8.254.65:37847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c266185732f43b3a5d9ad7543d628c9: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
I20260924 01:05:34.554389 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.538540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.8.254.66:40461)
0924 01:05:34.538726 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.538732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.538877 (+ 145us) server_negotiation.cc:407] Connection header received
0924 01:05:34.539132 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.539137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.539230 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.539351 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:34.541120 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.541962 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.543795 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.544320 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.544417 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.547717 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:34.547730 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:34.547745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:34.547780 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:34.551436 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.552200 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.552210 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.552214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.552385 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.552753 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.552758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.552761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.553214 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:34.553517 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.553741 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.553973 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:05:34.555202 22530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:34.593050 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.592460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56762)
0924 01:05:34.592726 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.592863 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 01:05:34.594358 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.593898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56772)
0924 01:05:34.594071 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.594181 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 01:05:34.646754 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22394
I20260924 01:05:34.655378 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.654758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56788)
0924 01:05:34.655067 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.655178 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260924 01:05:34.656642 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.656183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56790)
0924 01:05:34.656374 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.656471 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 01:05:34.665241 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22539
I20260924 01:05:34.682161 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.666117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.8.254.67:45859)
0924 01:05:34.666379 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:34.666385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:34.666406 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:05:34.666622 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:34.666627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:34.666735 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:34.666866 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:34.668268 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.669105 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.671340 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:34.671927 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:34.672032 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:34.675695 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:34.675709 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:34.675726 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:34.675767 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:34.679104 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.680003 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.680011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.680013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.680172 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:34.680584 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:34.680587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:34.680589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:34.680992 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:34.681239 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:34.681616 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 01:05:34.681821 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 01:05:34.682955 22664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:34.737517 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.736939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56798)
0924 01:05:34.737226 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.737338 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 01:05:34.738806 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.738320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56814)
0924 01:05:34.738545 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.738633 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 01:05:34.807013 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22539
I20260924 01:05:34.827577 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22160
I20260924 01:05:34.828943 22223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:34.941838 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:34.941103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37847 (local address 127.0.0.1:56818)
0924 01:05:34.941413 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:34.941496 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 01:05:34.941967 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22160
I20260924 01:05:34.964702 9209 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 01:05:35.300403 9209 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 01:05:35.300628 9209 mini_ranger.cc:66] Stopping Ranger...
I20260924 01:05:35.646152 9209 mini_ranger.cc:68] Stopped Ranger
2026-09-24 01:05:35.646 UTC [21211] LOG: received smart shutdown request
2026-09-24 01:05:35.649 UTC [21211] LOG: background worker "logical replication launcher" (PID 21220) exited with exit code 1
2026-09-24 01:05:35.649 UTC [21215] LOG: shutting down
2026-09-24 01:05:35.825 UTC [21211] LOG: database system is shut down
2026-09-24T01:05:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54408 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:05:36 dist-test-slave-5c7l krb5kdc[22736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:05:36 dist-test-slave-5c7l krb5kdc[22736](info): set up 2 sockets
Sep 24 01:05:36 dist-test-slave-5c7l krb5kdc[22736](info): commencing operation
krb5kdc: starting...
W20260924 01:05:37.063805 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:05:37 dist-test-slave-5c7l krb5kdc[22736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:05:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:05:37.301837 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44579
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:44579,127.8.254.125:38325,127.8.254.124:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:37.637246 22749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:37.637530 22749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:37.637627 22749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:37.644883 22749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:37.644994 22749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:37.645040 22749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:37.645089 22749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:37.645133 22749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:37.655361 22749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:44579,127.8.254.125:38325,127.8.254.124:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:37.656713 22749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:37.657961 22749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:37.670104 22754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:37.670140 22755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:37.670087 22757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:37.671051 22749 server_base.cc:1034] running on GCE node
I20260924 01:05:37.671859 22749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:37.673287 22749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:37.674528 22749 hybrid_clock.cc:648] HybridClock initialized: now 1790211937674478 us; error 45 us; skew 500 ppm
I20260924 01:05:37.674938 22749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:37 dist-test-slave-5c7l krb5kdc[22736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:37.678082 22749 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:37.679723 22749 webserver.cc:460] Webserver started at http://127.8.254.126:34077/ using document root <none> and password file <none>
I20260924 01:05:37.680438 22749 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:37.680603 22749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:37.680927 22749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:37.684258 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "88f34de26a1a4b96a93994aa74dc9ae9"
format_stamp: "Formatted at 2026-09-24 01:05:37 on dist-test-slave-5c7l"
server_key: "6426f077f6abb06288b0ed9ed21e44b0"
server_key_iv: "a0c37e62d3d85699203ca5ec5f5bad8c"
server_key_version: "encryptionkey@0"
I20260924 01:05:37.685245 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "88f34de26a1a4b96a93994aa74dc9ae9"
format_stamp: "Formatted at 2026-09-24 01:05:37 on dist-test-slave-5c7l"
server_key: "6426f077f6abb06288b0ed9ed21e44b0"
server_key_iv: "a0c37e62d3d85699203ca5ec5f5bad8c"
server_key_version: "encryptionkey@0"
I20260924 01:05:37.690877 22749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:05:37.694511 22764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:37.695966 22749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:05:37.696173 22749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "88f34de26a1a4b96a93994aa74dc9ae9"
format_stamp: "Formatted at 2026-09-24 01:05:37 on dist-test-slave-5c7l"
server_key: "6426f077f6abb06288b0ed9ed21e44b0"
server_key_iv: "a0c37e62d3d85699203ca5ec5f5bad8c"
server_key_version: "encryptionkey@0"
I20260924 01:05:37.696393 22749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:37.713287 22749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:37.717687 22749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:37.717984 22749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:37.736603 22749 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:44579
I20260924 01:05:37.736622 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:44579 every 8 connection(s)
I20260924 01:05:37.738559 22749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:37.742534 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:37.745443 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22749
I20260924 01:05:37.745700 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:05:37.746418 9209 external_mini_cluster.cc:1428] Setting key 4e0cda5ddc819a48a29ac7b4f8346e9a
I20260924 01:05:37.749506 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:05:37 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.125@KRBTEST.COM" created.
I20260924 01:05:37.773458 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.752880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.8.254.126:44579)
0924 01:05:37.754771 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:37.754784 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:37.754842 (+ 58us) server_negotiation.cc:407] Connection header received
0924 01:05:37.756406 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:37.756412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:37.756687 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:37.756904 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:37.758262 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:37.759486 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:37.760902 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:37.761471 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:37.761682 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:37.765364 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:37.765377 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:37.765395 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:37.765437 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:37.768816 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:37.769749 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:37.769757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:37.769759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:37.769931 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:37.770519 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:37.770523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:37.770525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:37.771136 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:37.771307 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:37.772021 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 01:05:37.772334 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1629,"thread_start_us":1380,"threads_started":1}
I20260924 01:05:37.773497 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.752881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44579 (local address 127.0.0.1:59396)
0924 01:05:37.753820 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.753875 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:37.756009 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:37.756942 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:37.756949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:37.757673 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:37.758014 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:37.758032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:37.759648 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:37.759654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:37.760758 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:37.760771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:37.761721 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:37.761728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:37.761889 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:37.762683 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:37.762727 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:37.765089 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:37.768974 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:37.768988 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:37.769003 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:37.769566 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:37.770108 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:37.770116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:37.770122 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:37.770393 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:37.771302 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:37.771313 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:37.771610 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 01:05:37.771890 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 01:05:37.772572 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":70,"thread_start_us":395,"threads_started":1}
I20260924 01:05:37.776546 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:05:37.777748 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.777358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48834)
0924 01:05:37.777531 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.777640 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 01:05:37.778120 22765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
I20260924 01:05:37.779676 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.779352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48846)
0924 01:05:37.779485 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.779548 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 01:05:37.779970 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:38325: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab.
Entry for principal kudu/127.8.254.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab.
I20260924 01:05:37.798308 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:38325
--webserver_interface=127.8.254.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab
--principal=kudu/127.8.254.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:44579,127.8.254.125:38325,127.8.254.124:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:37.829326 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } attempt: 1
I20260924 01:05:37.830619 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.830108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48850)
0924 01:05:37.830371 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.830488 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 01:05:37.831758 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.831398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48866)
0924 01:05:37.831533 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.831600 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 01:05:37.832031 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:38325: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
I20260924 01:05:37.900269 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } attempt: 2
I20260924 01:05:37.901535 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.901049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48882)
0924 01:05:37.901289 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.901390 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 01:05:37.902612 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.902277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48884)
0924 01:05:37.902408 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.902484 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 01:05:37.902858 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:38325: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
I20260924 01:05:37.994156 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } attempt: 3
I20260924 01:05:37.995419 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.994965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48900)
0924 01:05:37.995175 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.995285 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 01:05:37.996510 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:37.996148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48902)
0924 01:05:37.996306 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:37.996378 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 01:05:37.996711 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:38325: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
W20260924 01:05:38.131726 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:38.132020 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:38.132187 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:38.139555 22823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:38.139667 22823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:38.139762 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:38.139854 22823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:38.139922 22823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:38.139985 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } attempt: 4
I20260924 01:05:38.141245 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.140719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48908)
0924 01:05:38.141018 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.141105 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260924 01:05:38.142367 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.141941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48910)
0924 01:05:38.142083 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.142181 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 01:05:38.142585 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:38325: Network error: Client connection negotiation failed: client connection to 127.8.254.125:38325: connect: Connection refused (error 111)
I20260924 01:05:38.150790 22823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:44579,127.8.254.125:38325,127.8.254.124:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab
--principal=kudu/127.8.254.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:38325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.254.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:38.152459 22823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:38.154188 22823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:38.171181 22828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:38.171267 22829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:38.171294 22831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:38.172636 22823 server_base.cc:1034] running on GCE node
I20260924 01:05:38.173410 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:38.174947 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:38.176216 22823 hybrid_clock.cc:648] HybridClock initialized: now 1790211938176156 us; error 27 us; skew 500 ppm
I20260924 01:05:38.176612 22823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:38.187680 22823 init.cc:377] Logged in from keytab as kudu/127.8.254.125@KRBTEST.COM (short username kudu)
I20260924 01:05:38.189311 22823 webserver.cc:460] Webserver started at http://127.8.254.125:42609/ using document root <none> and password file <none>
I20260924 01:05:38.190033 22823 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:38.190186 22823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:38.190570 22823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:38.193548 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data/instance:
uuid: "f070d446b84a47858c9ce235c8ca5975"
format_stamp: "Formatted at 2026-09-24 01:05:38 on dist-test-slave-5c7l"
server_key: "6760b49d575437f619d82372ce32d850"
server_key_iv: "d4d5d7f93871793c05b2c75767eeec56"
server_key_version: "encryptionkey@0"
I20260924 01:05:38.194438 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal/instance:
uuid: "f070d446b84a47858c9ce235c8ca5975"
format_stamp: "Formatted at 2026-09-24 01:05:38 on dist-test-slave-5c7l"
server_key: "6760b49d575437f619d82372ce32d850"
server_key_iv: "d4d5d7f93871793c05b2c75767eeec56"
server_key_version: "encryptionkey@0"
I20260924 01:05:38.199504 22823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 01:05:38.203106 22838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:38.204460 22823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:05:38.204655 22823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal
uuid: "f070d446b84a47858c9ce235c8ca5975"
format_stamp: "Formatted at 2026-09-24 01:05:38 on dist-test-slave-5c7l"
server_key: "6760b49d575437f619d82372ce32d850"
server_key_iv: "d4d5d7f93871793c05b2c75767eeec56"
server_key_version: "encryptionkey@0"
I20260924 01:05:38.204859 22823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:38.230073 22823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:38.234426 22823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:38.234751 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:38.253777 22823 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.125:38325
I20260924 01:05:38.253799 22889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.125:38325 every 8 connection(s)
I20260924 01:05:38.255270 22823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
I20260924 01:05:38.259249 22890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:38.261725 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22823
I20260924 01:05:38.261956 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-1/wal/instance
I20260924 01:05:38.262523 9209 external_mini_cluster.cc:1428] Setting key 4d4a9eb77d7e1ddc33f20958e418f27a
I20260924 01:05:38.266476 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.124@KRBTEST.COM" created.
I20260924 01:05:38.287348 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.269215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.8.254.126:44579)
0924 01:05:38.269397 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.269402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.270980 (+ 1578us) server_negotiation.cc:407] Connection header received
0924 01:05:38.272862 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.272868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.272968 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.273087 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.274751 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.275439 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.277015 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.277444 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.277583 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.281559 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.281567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.281574 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.281617 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.284216 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.285007 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.285012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.285016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.285154 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.285567 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.285572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.285576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.285859 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.286016 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.286778 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.287009 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 01:05:38.288414 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.269954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44579 (local address 127.0.0.1:59404)
0924 01:05:38.270810 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.270852 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.272572 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.273353 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.273362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.273904 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.274568 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.274587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.275616 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.275620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.276865 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.276877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.277682 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.277690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.277916 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.278663 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.278686 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.281313 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.284374 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.284383 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.284401 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.284873 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.285266 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.285272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.285276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.285450 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.285991 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.285999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.286410 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.286679 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.287196 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
I20260924 01:05:38.290092 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:05:38.307683 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.290923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48914)
0924 01:05:38.291088 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.291108 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.291211 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.292487 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.292490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.292879 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.293024 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.293032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.295297 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.295301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.296143 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.296154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.297422 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.297426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.297596 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.298281 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.298297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.300555 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.305154 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.305159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.305161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.305553 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.305989 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.305993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.305995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.306135 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.306862 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.306866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.307026 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.307150 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.307365 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 01:05:38.307999 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.291227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.8.254.125:38325)
0924 01:05:38.291957 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.291967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.292000 (+ 33us) server_negotiation.cc:407] Connection header received
0924 01:05:38.292058 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.292068 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.292317 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.292475 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.293182 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.295128 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.296285 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.297279 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.297425 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.300843 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.300860 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.300882 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.300925 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.305013 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.305701 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.305713 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.305719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.305874 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.306284 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.306290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.306294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.306735 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.306925 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.307309 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.307574 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":374,"threads_started":1}
I20260924 01:05:38.310703 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:05:38.311942 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.311464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52720)
0924 01:05:38.311632 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.311790 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 01:05:38.312251 22842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.313804 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.313491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52724)
0924 01:05:38.313629 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.313707 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 01:05:38.314066 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab.
Entry for principal kudu/127.8.254.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab.
I20260924 01:05:38.317041 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:46883
--webserver_interface=127.8.254.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab
--principal=kudu/127.8.254.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:46501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:44579,127.8.254.125:38325,127.8.254.124:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:38.363417 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 1
I20260924 01:05:38.364773 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.364244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52738)
0924 01:05:38.364464 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.364604 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 01:05:38.365902 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.365501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52740)
0924 01:05:38.365719 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.365789 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 01:05:38.366180 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.434505 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 2
I20260924 01:05:38.435688 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.435225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52744)
0924 01:05:38.435429 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.435534 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 01:05:38.436676 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.436343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52750)
0924 01:05:38.436488 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.436563 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 01:05:38.436898 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.441869 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } attempt: 5
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:05:38.456884 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.442668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.8.254.125:38325)
0924 01:05:38.442846 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.442850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.442869 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:05:38.443095 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.443098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.443174 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.443257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.444441 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.445164 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.447117 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.447593 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.447706 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.451429 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.451434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.451437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.451503 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.454287 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.454965 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.454971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.454975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.455110 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.455490 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.455496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.455499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.455787 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.455958 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.456339 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.456553 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:05:38.456885 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.442553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48926)
0924 01:05:38.442784 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.442807 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.442956 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.443334 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.443339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.443955 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.444246 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.444260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.445314 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.445320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.446953 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.446974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.447776 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.447783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.447961 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.448612 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.448635 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.451229 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.454439 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.454445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.454449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.454839 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.455229 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.455234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.455238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.455379 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.455920 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.455925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.456089 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.456235 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.456530 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 01:05:38.457969 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:05:38.459251 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.458708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52752)
0924 01:05:38.458908 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.459024 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 01:05:38.460556 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.460061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52768)
0924 01:05:38.460268 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.460392 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 01:05:38.460836 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.512113 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 1
I20260924 01:05:38.513532 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.512872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52774)
0924 01:05:38.513134 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.513295 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 01:05:38.514710 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.514327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52784)
0924 01:05:38.514495 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.514581 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 01:05:38.514982 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.528110 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 3
I20260924 01:05:38.529292 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.528844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52786)
0924 01:05:38.529036 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.529129 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 01:05:38.530447 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.529963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52790)
0924 01:05:38.530232 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.530329 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260924 01:05:38.530652 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.583241 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 2
I20260924 01:05:38.584539 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.583927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52804)
0924 01:05:38.584220 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.584334 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 01:05:38.585637 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.585219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52820)
0924 01:05:38.585393 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.585491 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 01:05:38.585865 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
W20260924 01:05:38.651075 22897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:38.651503 22897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:38.651679 22897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:38.659134 22897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:38.659273 22897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:38.659354 22897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:38.659458 22897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:38.659541 22897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:38.669588 22897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:46501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:44579,127.8.254.125:38325,127.8.254.124:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab
--principal=kudu/127.8.254.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.254.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:38.671032 22897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:38.672375 22897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 01:05:38.673867 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 4
I20260924 01:05:38.674978 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.674491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52826)
0924 01:05:38.674731 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.674816 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 01:05:38.675976 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.675640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52828)
0924 01:05:38.675803 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.675873 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 01:05:38.676184 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
W20260924 01:05:38.684317 22903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:38.684361 22902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:38.684404 22905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:38.685484 22897 server_base.cc:1034] running on GCE node
I20260924 01:05:38.686482 22897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:38.687927 22897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:38.689157 22897 hybrid_clock.cc:648] HybridClock initialized: now 1790211938689100 us; error 52 us; skew 500 ppm
I20260924 01:05:38.689512 22897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:38.692076 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 3
I20260924 01:05:38.692893 22897 init.cc:377] Logged in from keytab as kudu/127.8.254.124@KRBTEST.COM (short username kudu)
I20260924 01:05:38.693295 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.692726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52830)
0924 01:05:38.692940 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.693045 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 01:05:38.694473 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.694069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52842)
0924 01:05:38.694311 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.694378 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 01:05:38.694669 22816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:46883: Network error: Client connection negotiation failed: client connection to 127.8.254.124:46883: connect: Connection refused (error 111)
I20260924 01:05:38.695031 22897 webserver.cc:460] Webserver started at http://127.8.254.124:34147/ using document root <none> and password file <none>
I20260924 01:05:38.695747 22897 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:38.695906 22897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:38.696228 22897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:38.699307 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data/instance:
uuid: "65598e6c894946aeab65ed6e62e8bdc1"
format_stamp: "Formatted at 2026-09-24 01:05:38 on dist-test-slave-5c7l"
server_key: "9b682e18ca397d21c2a6748ad9f31fd6"
server_key_iv: "ba73fe0d5b5e1f1a11265b90dd1ac8ab"
server_key_version: "encryptionkey@0"
I20260924 01:05:38.700174 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal/instance:
uuid: "65598e6c894946aeab65ed6e62e8bdc1"
format_stamp: "Formatted at 2026-09-24 01:05:38 on dist-test-slave-5c7l"
server_key: "9b682e18ca397d21c2a6748ad9f31fd6"
server_key_iv: "ba73fe0d5b5e1f1a11265b90dd1ac8ab"
server_key_version: "encryptionkey@0"
I20260924 01:05:38.705493 22897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:05:38.709187 22912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:38.710785 22897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:05:38.710973 22897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal
uuid: "65598e6c894946aeab65ed6e62e8bdc1"
format_stamp: "Formatted at 2026-09-24 01:05:38 on dist-test-slave-5c7l"
server_key: "9b682e18ca397d21c2a6748ad9f31fd6"
server_key_iv: "ba73fe0d5b5e1f1a11265b90dd1ac8ab"
server_key_version: "encryptionkey@0"
I20260924 01:05:38.711174 22897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:38.748365 22897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:38.752800 22897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:38.753098 22897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:38.772207 22897 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.124:46883
I20260924 01:05:38.772248 22963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.124:46883 every 8 connection(s)
I20260924 01:05:38.773708 22897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
I20260924 01:05:38.777438 22964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:38.780069 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 22897
I20260924 01:05:38.780288 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790211722208042-9209-0/minicluster-data/master-2/wal/instance
I20260924 01:05:38.780870 9209 external_mini_cluster.cc:1428] Setting key b1420432e013570be88c5ea0f3d935fc
I20260924 01:05:38.786043 22964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:38.806663 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.789108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.8.254.126:44579)
0924 01:05:38.790443 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.790449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.790468 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:05:38.790525 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.790530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.790602 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.790724 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.792181 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.793434 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.795446 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.795879 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.795982 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.799702 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.799730 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.799737 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.799782 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.803371 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.804214 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.804218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.804220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.804367 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.804765 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.804770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.804772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.805093 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.805271 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.806001 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.806279 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":1106,"threads_started":1}
I20260924 01:05:38.807668 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.789090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:44579 (local address 127.0.0.1:59410)
0924 01:05:38.789930 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.789985 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.790174 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.790786 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.790791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.791638 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.791989 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.792003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.793633 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.793637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.795275 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.795294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.796137 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.796146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.796364 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.797063 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.797101 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.799474 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.803510 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.803523 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.803538 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.804076 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.804485 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.804489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.804491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.804638 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.805237 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.805243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.805496 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.805776 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.806596 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":283,"thread_start_us":302,"threads_started":1}
I20260924 01:05:38.807749 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.782963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.8.254.124:46883)
0924 01:05:38.784138 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.784150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.784194 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:05:38.785810 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.785837 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.786321 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.786671 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.788603 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.790742 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.792561 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.793758 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.793905 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.798928 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.798937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.798951 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.798990 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.802949 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.803787 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.803794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.803796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.803957 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.804412 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.804419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.804423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.805215 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.805452 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.806013 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.806360 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":640,"threads_started":1}
I20260924 01:05:38.810750 22964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:38.825774 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.811554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:38325 (local address 127.0.0.1:48936)
0924 01:05:38.811711 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.811725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.811832 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.812459 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.812463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.812830 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.812974 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.812982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.814395 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.814399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.815791 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.815807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.816575 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.816579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.816731 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.817318 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.817336 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.819465 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.823380 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.823384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.823387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.823746 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.824191 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.824199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.824202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.824337 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.824868 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.824875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.825045 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.825175 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.825479 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 01:05:38.825947 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.811947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.8.254.125:38325)
0924 01:05:38.812143 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.812151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.812169 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:05:38.812226 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.812231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.812301 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.812425 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.813115 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.814254 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.815924 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.816436 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.816560 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.819656 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.819663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.819668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.819714 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.823237 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.823877 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.823885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.823889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.824064 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.824468 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.824473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.824476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.824742 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.824856 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.825308 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.825508 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:05:38.827030 22964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:05:38.827205 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.811457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.8.254.126:44579)
0924 01:05:38.811923 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.811933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.811946 (+ 13us) server_negotiation.cc:407] Connection header received
0924 01:05:38.812006 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.812013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.812100 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.812210 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.813389 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.814479 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.816283 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.816870 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.817004 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.821139 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.821145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.821150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.821191 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.824327 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.825050 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.825054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.825056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.825193 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.825598 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.825603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.825607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.825968 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.826283 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.826651 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.826895 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:05:38.840657 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.827969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52866)
0924 01:05:38.828095 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.828112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.828228 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.828510 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.828514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.829005 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.829196 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.829209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.830306 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.830312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.831385 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.831396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.832185 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.832191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.832393 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.832976 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.832999 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.835157 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.838151 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.838159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.838164 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.838555 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.838917 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.838925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.838929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.839055 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.839584 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.839588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.839731 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.839883 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.840195 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 01:05:38.840684 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.827904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.8.254.124:46883)
0924 01:05:38.828094 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.828101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.828166 (+ 65us) server_negotiation.cc:407] Connection header received
0924 01:05:38.828336 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.828341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.828401 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.828496 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.829342 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.830114 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.831525 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.832038 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.832129 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.835366 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.835371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.835374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.835401 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.838022 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.838670 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.838677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.838681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.838807 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.839166 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.839171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.839174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.839459 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.839604 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.839980 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.840359 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":64}
I20260924 01:05:38.843961 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.828924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.8.254.125:38325)
0924 01:05:38.829078 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.829082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.829097 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:05:38.829296 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.829301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.829374 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.829480 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.830848 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.831607 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.833184 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.833609 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.833706 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.838262 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.838271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.838281 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.838313 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.840994 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.841817 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.841823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.841826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.841996 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.842477 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.842483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.842486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.842765 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.843066 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.843387 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.843626 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:05:38.846674 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: Bootstrap starting.
I20260924 01:05:38.848898 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:05:38.850183 22964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:38.851300 22964 log.cc:824] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:38.854430 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: No bootstrap required, opened a new log
I20260924 01:05:38.869401 22964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:38.870949 22964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:38.871168 22964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65598e6c894946aeab65ed6e62e8bdc1, State: Initialized, Role: FOLLOWER
I20260924 01:05:38.871855 22816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 4
I20260924 01:05:38.874289 22964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:38.878058 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:38.878640 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:38.879879 22979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:05:38.879976 22964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:38.880251 22979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:38.884127 22979 leader_election.cc:288] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88f34de26a1a4b96a93994aa74dc9ae9 (127.8.254.126:44579), f070d446b84a47858c9ce235c8ca5975 (127.8.254.125:38325)
I20260924 01:05:38.891340 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.854007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.8.254.125:38325)
0924 01:05:38.854227 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.854243 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.866620 (+ 12377us) server_negotiation.cc:407] Connection header received
0924 01:05:38.866949 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.866958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.867051 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.867130 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.869603 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.870671 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.872824 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.873447 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.873540 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.882432 (+ 8892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.882444 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.882452 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.882512 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.887717 (+ 5205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.888546 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.888553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.888557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.888744 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.889277 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.889284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.889290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.889707 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.890002 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.890687 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.890958 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211937, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
I20260924 01:05:38.895942 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.858764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.8.254.124:46883)
0924 01:05:38.858975 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.858982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.870684 (+ 11702us) server_negotiation.cc:407] Connection header received
0924 01:05:38.870877 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.870882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.870967 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.871066 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.873443 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.874936 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.877514 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.879346 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.879478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.883047 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.883053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.883058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.883096 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.892348 (+ 9252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.893278 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.893283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.893286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.893427 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.893903 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.893910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.893915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.894351 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.894593 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.895257 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.895580 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 01:05:38.900053 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.854356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.8.254.126:44579)
0924 01:05:38.854562 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.854568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.864729 (+ 10161us) server_negotiation.cc:407] Connection header received
0924 01:05:38.864913 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.864923 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.865043 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.865184 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.869212 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.870396 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.872686 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.873352 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.873448 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.883558 (+ 10110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.883568 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.883576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.883627 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.889255 (+ 5628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.896235 (+ 6980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.896241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.896262 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.896434 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.898410 (+ 1976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.898415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.898419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.898795 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.899044 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.899356 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.899648 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:05:38.906035 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.872471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52882)
0924 01:05:38.872878 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.872897 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.873006 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.878318 (+ 5312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.878325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.878852 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.879053 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.879061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.886425 (+ 7364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.886434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.888481 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.888503 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.890741 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.890752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.891517 (+ 765us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.892538 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.892563 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.896793 (+ 4230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.902572 (+ 5779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.902580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.902587 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.903495 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.904075 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.904080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.904084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.904261 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.905001 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.905022 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.905190 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.905345 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.905628 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260924 01:05:38.906133 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.873244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.8.254.124:46883)
0924 01:05:38.877336 (+ 4092us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.877343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.877368 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:05:38.877419 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.877427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.877502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.877636 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.879252 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.886148 (+ 6896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.888825 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.889811 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.889941 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.897092 (+ 7151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.897097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.897101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.897138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.901698 (+ 4560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.903716 (+ 2018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.903724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.903730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.903939 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.904389 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.904396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.904402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.904880 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.905070 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.905478 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.905684 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3950,"thread_start_us":3727,"threads_started":1}
I20260924 01:05:38.913806 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: Bootstrap starting.
I20260924 01:05:38.918001 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:38.920910 22816 log.cc:824] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:38.924590 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: No bootstrap required, opened a new log
W20260924 01:05:38.933017 22990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:05:38.933182 22990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:05:38.938112 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:38.938578 22816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:38.938699 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f34de26a1a4b96a93994aa74dc9ae9, State: Initialized, Role: FOLLOWER
I20260924 01:05:38.939386 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:38.917337 22897 master_runner.cc:186] Error getting master registration for 127.8.254.126:44579: OK, instance_id { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" instance_seqno: 1790211937731130 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 01:05:38.940582 22897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 01:05:38.940475 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:38.940996 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:38.941731 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:38.941829 22791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65598e6c894946aeab65ed6e62e8bdc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" is_pre_election: true
I20260924 01:05:38.942552 22791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65598e6c894946aeab65ed6e62e8bdc1 in term 0.
I20260924 01:05:38.943464 22914 leader_election.cc:302] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65598e6c894946aeab65ed6e62e8bdc1, 88f34de26a1a4b96a93994aa74dc9ae9; no voters:
I20260924 01:05:38.944052 22979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:05:38.944212 22979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:05:38.944339 22979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:38.947893 22979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:38.948693 22979 leader_election.cc:288] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [CANDIDATE]: Term 1 election: Requested vote from peers 88f34de26a1a4b96a93994aa74dc9ae9 (127.8.254.126:44579), f070d446b84a47858c9ce235c8ca5975 (127.8.254.125:38325)
I20260924 01:05:38.950501 22791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65598e6c894946aeab65ed6e62e8bdc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88f34de26a1a4b96a93994aa74dc9ae9"
I20260924 01:05:38.950752 22791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:38.954901 22791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65598e6c894946aeab65ed6e62e8bdc1 in term 1.
I20260924 01:05:38.955478 22914 leader_election.cc:302] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65598e6c894946aeab65ed6e62e8bdc1, 88f34de26a1a4b96a93994aa74dc9ae9; no voters:
I20260924 01:05:38.955864 22979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:38.962873 22979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [term 1 LEADER]: Becoming Leader. State: Replica: 65598e6c894946aeab65ed6e62e8bdc1, State: Running, Role: LEADER
I20260924 01:05:38.963759 22979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
W20260924 01:05:38.970976 22770 tablet.cc:2369] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:38.971242 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65598e6c894946aeab65ed6e62e8bdc1. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:38.971802 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:38.973384 23008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:38.975395 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } attempt: 5
I20260924 01:05:38.978138 23008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 01:05:38.984913 23005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:05:38.985056 23005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 01:05:38 dist-test-slave-5c7l krb5kdc[22736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211938, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
I20260924 01:05:38.973914 22749 master_runner.cc:186] Error getting master registration for 127.8.254.125:38325: OK, instance_id { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" instance_seqno: 1790211938248194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 01:05:38.987546 22749 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 01:05:38.996429 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.976048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:46883 (local address 127.0.0.1:52890)
0924 01:05:38.976447 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:38.976467 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:38.976588 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:38.977597 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:38.977606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:38.978217 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:38.978449 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.978461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.980012 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.980017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.981449 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:38.981465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.982675 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.982681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:38.982971 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.983692 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:38.983715 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:38.986941 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:38.992384 (+ 5443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.992394 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.992401 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.993148 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.993732 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:38.993740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:38.993747 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:38.993976 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:38.994962 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:38.994970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:38.995189 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 01:05:38.995391 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 01:05:38.995838 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260924 01:05:38.997910 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:38.976911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.8.254.124:46883)
0924 01:05:38.977087 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:38.977093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:38.977112 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:05:38.977177 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:38.977182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:38.977289 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:38.977388 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:38.978711 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.979839 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.981633 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:38.982320 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:38.982445 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:38.987200 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:38.987209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:38.987215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:38.987270 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:38.992146 (+ 4876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.993340 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.993349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.993355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.993540 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:38.994138 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:38.994146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:38.994151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:38.994655 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:38.994877 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:38.997039 (+ 2162us) server_negotiation.cc:299] Negotiation successful
0924 01:05:38.997397 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 01:05:39.005353 22918 tablet.cc:2369] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:39.014176 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: Bootstrap starting.
I20260924 01:05:39.019882 22890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:39.021827 22890 log.cc:824] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:39.034981 22791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [term 1 FOLLOWER]: Refusing update from remote peer 65598e6c894946aeab65ed6e62e8bdc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:39.036073 22979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:39.038664 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: No bootstrap required, opened a new log
I20260924 01:05:39.055225 22890 raft_consensus.cc:348] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:39.055706 22890 raft_consensus.cc:374] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:39.055835 22890 raft_consensus.cc:729] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f070d446b84a47858c9ce235c8ca5975, State: Initialized, Role: FOLLOWER
I20260924 01:05:39.056514 22890 consensus_queue.cc:260] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } }
I20260924 01:05:39.057507 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.057888 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:39.059432 22890 sys_catalog.cc:564] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:39.059479 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65598e6c894946aeab65ed6e62e8bdc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f070d446b84a47858c9ce235c8ca5975"
I20260924 01:05:39.059993 22864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:39.062453 22865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65598e6c894946aeab65ed6e62e8bdc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f070d446b84a47858c9ce235c8ca5975" is_pre_election: true
I20260924 01:05:39.066476 22864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65598e6c894946aeab65ed6e62e8bdc1 in term 1.
I20260924 01:05:39.067327 22863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [term 1 FOLLOWER]: Refusing update from remote peer 65598e6c894946aeab65ed6e62e8bdc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:39.069171 22979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:39.082271 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65598e6c894946aeab65ed6e62e8bdc1. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.082643 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:39.092351 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65598e6c894946aeab65ed6e62e8bdc1. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.092653 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:39.094870 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.095125 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:39.095512 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.095744 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:39.100073 23008 catalog_manager.cc:1347] Generated new cluster ID: 2e096d89f49a4bbba12114f43ff7b2a6
I20260924 01:05:39.100169 23008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:05:39.110597 23016 mvcc.cc:205] Tried to move back new op lower bound from 7332708102276050944 to 7332708102001164288. Current Snapshot: MvccSnapshot[applied={T|T < 7332708102276050944}]
I20260924 01:05:39.112285 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.112527 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:39.122277 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65598e6c894946aeab65ed6e62e8bdc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f34de26a1a4b96a93994aa74dc9ae9" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 44579 } } peers { permanent_uuid: "f070d446b84a47858c9ce235c8ca5975" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 38325 } } peers { permanent_uuid: "65598e6c894946aeab65ed6e62e8bdc1" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 46883 } } }
I20260924 01:05:39.122645 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:05:39.164675 23008 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:05:39.166855 23008 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:05:39.189086 23008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65598e6c894946aeab65ed6e62e8bdc1: Generated new TSK 0
I20260924 01:05:39.190374 23008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:05:39.193428 23030 catalog_manager.cc:1259] Loaded cluster ID: 2e096d89f49a4bbba12114f43ff7b2a6
I20260924 01:05:39.193523 23030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: loading cluster ID for follower catalog manager: success
I20260924 01:05:39.197223 23030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: acquiring CA information for follower catalog manager: success
I20260924 01:05:39.199677 23030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f070d446b84a47858c9ce235c8ca5975: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 01:05:39.987157 23005 catalog_manager.cc:1259] Loaded cluster ID: 2e096d89f49a4bbba12114f43ff7b2a6
I20260924 01:05:39.987275 23005 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: loading cluster ID for follower catalog manager: success
I20260924 01:05:39.989859 23005 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: acquiring CA information for follower catalog manager: success
I20260924 01:05:39.991761 23005 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88f34de26a1a4b96a93994aa74dc9ae9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 01:05:40.320384 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22749
I20260924 01:05:40.321717 22811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:40.431752 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22749
I20260924 01:05:40.453150 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22823
I20260924 01:05:40.454428 22885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:40.563373 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22823
I20260924 01:05:40.584205 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 22897
I20260924 01:05:40.585331 22959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:40.695423 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 22897
2026-09-24T01:05:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4747 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:05:40 dist-test-slave-5c7l krb5kdc[23040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:05:40 dist-test-slave-5c7l krb5kdc[23040](info): set up 2 sockets
Sep 24 01:05:40 dist-test-slave-5c7l krb5kdc[23040](info): commencing operation
krb5kdc: starting...
W20260924 01:05:41.823861 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:05:41 dist-test-slave-5c7l krb5kdc[23040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:05:42.063966 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:36119
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:36119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:42.399822 23053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:42.400179 23053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:42.400362 23053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:42.407652 23053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:42.407811 23053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:42.407932 23053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:42.408051 23053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:42.408135 23053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:42.418179 23053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:36119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:36119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:42.419535 23053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:42.420897 23053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:42.433065 23058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:42.433035 23059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:42.433043 23061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:42.433995 23053 server_base.cc:1034] running on GCE node
I20260924 01:05:42.434907 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:42.436447 23053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:42.437685 23053 hybrid_clock.cc:648] HybridClock initialized: now 1790211942437643 us; error 37 us; skew 500 ppm
I20260924 01:05:42.438059 23053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:42 dist-test-slave-5c7l krb5kdc[23040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211942, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:42.441262 23053 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:42.442852 23053 webserver.cc:460] Webserver started at http://127.8.254.126:35715/ using document root <none> and password file <none>
I20260924 01:05:42.443516 23053 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:42.443622 23053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:42.443898 23053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:42.446916 23053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "809e00a3c6d24b38b353ca256bd03041"
format_stamp: "Formatted at 2026-09-24 01:05:42 on dist-test-slave-5c7l"
server_key: "85ca6acc6ab642f44b733dedd55f9903"
server_key_iv: "becf6405441500870669377a37cb2238"
server_key_version: "encryptionkey@0"
I20260924 01:05:42.447639 23053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "809e00a3c6d24b38b353ca256bd03041"
format_stamp: "Formatted at 2026-09-24 01:05:42 on dist-test-slave-5c7l"
server_key: "85ca6acc6ab642f44b733dedd55f9903"
server_key_iv: "becf6405441500870669377a37cb2238"
server_key_version: "encryptionkey@0"
I20260924 01:05:42.452920 23053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:05:42.456432 23068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:42.457782 23053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:05:42.457958 23053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "809e00a3c6d24b38b353ca256bd03041"
format_stamp: "Formatted at 2026-09-24 01:05:42 on dist-test-slave-5c7l"
server_key: "85ca6acc6ab642f44b733dedd55f9903"
server_key_iv: "becf6405441500870669377a37cb2238"
server_key_version: "encryptionkey@0"
I20260924 01:05:42.458174 23053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:42.477619 23053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:42.481652 23053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:42.481909 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:42.501009 23053 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:36119
I20260924 01:05:42.501029 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:36119 every 8 connection(s)
I20260924 01:05:42.502496 23053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:42.506301 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:42.507206 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23053
I20260924 01:05:42.507462 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:05:42.508335 9209 external_mini_cluster.cc:1428] Setting key afe040e6409c68de615917c7ff75b329
I20260924 01:05:42.519618 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041: Bootstrap starting.
I20260924 01:05:42.522624 23120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041: Neither blocks nor log segments found. Creating new log.
Sep 24 01:05:42 dist-test-slave-5c7l krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:42.523671 23120 log.cc:824] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:42.527278 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041: No bootstrap required, opened a new log
I20260924 01:05:42.532704 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:42.510406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.8.254.126:36119)
0924 01:05:42.511153 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:42.511166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:42.511204 (+ 38us) server_negotiation.cc:407] Connection header received
0924 01:05:42.513167 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:42.513194 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:42.513611 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:42.514022 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:42.515800 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:42.517111 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:42.518503 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:42.519160 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:42.519374 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:42.523805 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:42.523818 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:42.523837 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:42.523889 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:42.528035 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:42.528815 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:42.528827 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:42.528837 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:42.529054 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:42.529428 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:42.529439 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:42.529443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:42.530163 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:42.530543 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:42.530835 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 01:05:42.531289 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":328,"threads_started":1}
I20260924 01:05:42.537020 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:05:42.537453 23120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809e00a3c6d24b38b353ca256bd03041" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36119 } }
I20260924 01:05:42.537827 23120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:42.537951 23120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809e00a3c6d24b38b353ca256bd03041, State: Initialized, Role: FOLLOWER
I20260924 01:05:42.538575 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809e00a3c6d24b38b353ca256bd03041" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36119 } }
I20260924 01:05:42.538825 23120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:05:42.538978 23120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:05:42.539147 23120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:42.541651 23120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809e00a3c6d24b38b353ca256bd03041" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36119 } }
I20260924 01:05:42.542157 23120 leader_election.cc:302] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809e00a3c6d24b38b353ca256bd03041; no voters:
I20260924 01:05:42.542821 23120 leader_election.cc:288] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:05:42.542982 23125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:42.543753 23125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [term 1 LEADER]: Becoming Leader. State: Replica: 809e00a3c6d24b38b353ca256bd03041, State: Running, Role: LEADER
I20260924 01:05:42.544294 23125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809e00a3c6d24b38b353ca256bd03041" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36119 } }
I20260924 01:05:42.544955 23120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:42.547379 23127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809e00a3c6d24b38b353ca256bd03041. Latest consensus state: current_term: 1 leader_uuid: "809e00a3c6d24b38b353ca256bd03041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809e00a3c6d24b38b353ca256bd03041" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36119 } } }
I20260924 01:05:42.547520 23126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809e00a3c6d24b38b353ca256bd03041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809e00a3c6d24b38b353ca256bd03041" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 36119 } } }
I20260924 01:05:42.547797 23127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:42.547883 23126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:42.550185 23133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:42.556144 23133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:05:42.565785 23133 catalog_manager.cc:1347] Generated new cluster ID: c61d1af1d3004f40a21a5a14583bb945
I20260924 01:05:42.565932 23133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:05:42.578989 23133 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:05:42.580886 23133 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:05:42.593030 23133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 809e00a3c6d24b38b353ca256bd03041: Generated new TSK 0
I20260924 01:05:42.593927 23133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:05:42.650128 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23053
I20260924 01:05:42.651467 23115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:42.749753 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23053
2026-09-24T01:05:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2049 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:05:42 dist-test-slave-5c7l krb5kdc[23148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:05:42 dist-test-slave-5c7l krb5kdc[23148](info): set up 2 sockets
Sep 24 01:05:42 dist-test-slave-5c7l krb5kdc[23148](info): commencing operation
krb5kdc: starting...
W20260924 01:05:43.875932 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:05:44 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:05:44.118165 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40049
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:44.451578 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:44.451897 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:44.452029 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:44.459494 23161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:44.459659 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:44.459753 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:44.459849 23161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:44.459913 23161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:44.470170 23161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:40049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:44.471565 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:44.472906 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:44.484935 23166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:44.484977 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:44.484933 23167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:44.485602 23161 server_base.cc:1034] running on GCE node
I20260924 01:05:44.486502 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:44.488605 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:44.489873 23161 hybrid_clock.cc:648] HybridClock initialized: now 1790211944489801 us; error 52 us; skew 500 ppm
I20260924 01:05:44.490404 23161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:44 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211944, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:44.495388 23161 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:44.497414 23161 webserver.cc:460] Webserver started at http://127.8.254.126:37663/ using document root <none> and password file <none>
I20260924 01:05:44.498361 23161 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:44.498507 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:44.498900 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:44.503005 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "76f1762dc09442f4a565fe4410eebc46"
format_stamp: "Formatted at 2026-09-24 01:05:44 on dist-test-slave-5c7l"
server_key: "7d9921ebedd97e941df0fe6bbf29fc9f"
server_key_iv: "4b57c40e95303d115aacf7a724f033c9"
server_key_version: "encryptionkey@0"
I20260924 01:05:44.503791 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "76f1762dc09442f4a565fe4410eebc46"
format_stamp: "Formatted at 2026-09-24 01:05:44 on dist-test-slave-5c7l"
server_key: "7d9921ebedd97e941df0fe6bbf29fc9f"
server_key_iv: "4b57c40e95303d115aacf7a724f033c9"
server_key_version: "encryptionkey@0"
I20260924 01:05:44.510927 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 01:05:44.515651 23176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:44.517194 23161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 01:05:44.517387 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "76f1762dc09442f4a565fe4410eebc46"
format_stamp: "Formatted at 2026-09-24 01:05:44 on dist-test-slave-5c7l"
server_key: "7d9921ebedd97e941df0fe6bbf29fc9f"
server_key_iv: "4b57c40e95303d115aacf7a724f033c9"
server_key_version: "encryptionkey@0"
I20260924 01:05:44.517594 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:44.535650 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:44.539755 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:44.540062 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:44.559149 23161 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:40049
I20260924 01:05:44.559171 23227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:40049 every 8 connection(s)
I20260924 01:05:44.560573 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:44.561111 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23161
I20260924 01:05:44.561349 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:05:44.562184 9209 external_mini_cluster.cc:1428] Setting key 57b30bc1c7f354be37dad4419503d6b5
I20260924 01:05:44.565230 23228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 01:05:44 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:44.579447 23228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap starting.
I20260924 01:05:44.583204 23228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:44.584321 23228 log.cc:824] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:44.587110 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:44.564413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.8.254.126:40049)
0924 01:05:44.565325 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:44.565343 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:44.565396 (+ 53us) server_negotiation.cc:407] Connection header received
0924 01:05:44.567094 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:44.567118 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:44.567503 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:44.567856 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:44.569666 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:44.571143 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:44.572994 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:44.573673 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:44.573833 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:44.578984 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:44.578992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:44.579009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:44.579054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:44.582136 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:44.583094 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:44.583102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:44.583108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:44.583339 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:44.583768 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:44.583775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:44.583778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:44.584313 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:44.584646 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:44.585176 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 01:05:44.585646 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":456,"threads_started":1}
I20260924 01:05:44.587857 23228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: No bootstrap required, opened a new log
I20260924 01:05:44.595984 23228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:44.596403 23228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:44.596529 23228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f1762dc09442f4a565fe4410eebc46, State: Initialized, Role: FOLLOWER
I20260924 01:05:44.597172 23228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:44.597450 23228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:05:44.597610 23228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:05:44.597808 23228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:44.600140 23228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:44.600682 23228 leader_election.cc:302] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f1762dc09442f4a565fe4410eebc46; no voters:
I20260924 01:05:44.601310 23228 leader_election.cc:288] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:05:44.601467 23233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:44.602632 23233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 LEADER]: Becoming Leader. State: Replica: 76f1762dc09442f4a565fe4410eebc46, State: Running, Role: LEADER
I20260924 01:05:44.603157 23233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:44.603633 23228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:44.606025 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f1762dc09442f4a565fe4410eebc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } } }
I20260924 01:05:44.606545 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:44.606146 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f1762dc09442f4a565fe4410eebc46. Latest consensus state: current_term: 1 leader_uuid: "76f1762dc09442f4a565fe4410eebc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } } }
I20260924 01:05:44.606887 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:44.607632 23242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:44.612455 23242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:05:44.621490 23242 catalog_manager.cc:1347] Generated new cluster ID: 00c7990f408246f8af76f10c44345303
I20260924 01:05:44.621618 23242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:05:44.646502 23242 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:05:44.648382 23242 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:05:44.658138 23242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Generated new TSK 0
I20260924 01:05:44.659075 23242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:05:44.722721 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40049
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:45.074100 23254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:45.074492 23254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:45.074592 23254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:45.081830 23254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:45.081933 23254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:45.082047 23254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:05:45.091756 23254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:45.092999 23254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:45.094370 23254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:45.107105 23260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:45.107172 23262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:45.107086 23259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:45.108397 23254 server_base.cc:1034] running on GCE node
I20260924 01:05:45.109054 23254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:45.109967 23254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:45.111224 23254 hybrid_clock.cc:648] HybridClock initialized: now 1790211945111131 us; error 71 us; skew 500 ppm
I20260924 01:05:45.111547 23254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:45 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:45.114636 23254 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:05:45.116005 23254 webserver.cc:460] Webserver started at http://127.8.254.65:43379/ using document root <none> and password file <none>
I20260924 01:05:45.116698 23254 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:45.116839 23254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:45.117147 23254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:45.120090 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "cd823dd8f79cbeac965db70a6f454b12"
server_key_iv: "b49aa0d682d2112b471b199e3de55dba"
server_key_version: "encryptionkey@0"
I20260924 01:05:45.120864 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "cd823dd8f79cbeac965db70a6f454b12"
server_key_iv: "b49aa0d682d2112b471b199e3de55dba"
server_key_version: "encryptionkey@0"
I20260924 01:05:45.126432 23254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:05:45.130074 23269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:45.131532 23254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:05:45.131726 23254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "cd823dd8f79cbeac965db70a6f454b12"
server_key_iv: "b49aa0d682d2112b471b199e3de55dba"
server_key_version: "encryptionkey@0"
I20260924 01:05:45.131954 23254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:45.155623 23254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:45.160456 23254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:45.160797 23254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:45.161839 23254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:45.163348 23254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:45.163455 23254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:45.163604 23254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:45.163684 23254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:45.199064 23254 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:35361
I20260924 01:05:45.199159 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:35361 every 8 connection(s)
I20260924 01:05:45.200417 23254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:05:45.210767 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23254
I20260924 01:05:45.211000 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:05:45.211652 9209 external_mini_cluster.cc:1428] Setting key e7a817f2ddb69486bc779d20456f6138
Sep 24 01:05:45 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:45.220975 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:45.203113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:53561 (local address 127.8.254.126:40049)
0924 01:05:45.203602 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:45.203607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:45.204667 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 01:05:45.206668 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:45.206675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:45.206774 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:45.206893 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:45.208487 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.209224 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.210855 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.211357 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.211502 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:45.215043 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:45.215048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:45.215052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:45.215083 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:45.217710 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:45.218604 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:45.218609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:45.218619 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:45.218741 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:45.219189 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:45.219193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:45.219195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:45.219465 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:45.219625 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:45.220368 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 01:05:45.220597 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260924 01:05:45.222012 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:45.203709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:53561)
0924 01:05:45.204499 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:45.204555 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:45.206347 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:45.207147 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:45.207155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:45.207729 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:45.208311 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:45.208331 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.209390 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.209394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:45.210697 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:45.210711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.211607 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.211614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:45.211905 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:45.212594 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:45.212616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:45.214812 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:45.217927 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:45.217937 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:45.217955 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:45.218476 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:45.218860 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:45.218864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:45.218868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:45.219058 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:45.219605 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:45.219614 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:45.219958 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 01:05:45.220235 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 01:05:45.220765 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":305,"threads_started":1}
I20260924 01:05:45.223663 23382 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40049
I20260924 01:05:45.224007 23382 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:45.224875 23382 heartbeater.cc:511] Master 127.8.254.126:40049 requested a full tablet report, sending...
I20260924 01:05:45.227378 23193 ts_manager.cc:194] Registered new tserver with Master: db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361)
I20260924 01:05:45.229594 23193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:53561
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:05:45.266145 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40049
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:45.604724 23388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:45.605017 23388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:45.605111 23388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:45.612601 23388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:45.612727 23388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:45.612843 23388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:05:45.622969 23388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:45.624239 23388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:45.625531 23388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:45.639576 23394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:45.639633 23396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:45.639544 23393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:45.640620 23388 server_base.cc:1034] running on GCE node
I20260924 01:05:45.641232 23388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:45.642246 23388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:45.643465 23388 hybrid_clock.cc:648] HybridClock initialized: now 1790211945643396 us; error 64 us; skew 500 ppm
I20260924 01:05:45.643805 23388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:45 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:45.647022 23388 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:05:45.648558 23388 webserver.cc:460] Webserver started at http://127.8.254.66:34359/ using document root <none> and password file <none>
I20260924 01:05:45.649304 23388 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:45.649458 23388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:45.649787 23388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:45.653771 23388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "a9a578d3cbc147e29b5e6f604723805c"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "d109b661c5f7b288c92a85f3a68e1682"
server_key_iv: "d67c9c5bd75e0baa27b9c3d6359223a4"
server_key_version: "encryptionkey@0"
I20260924 01:05:45.654809 23388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "a9a578d3cbc147e29b5e6f604723805c"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "d109b661c5f7b288c92a85f3a68e1682"
server_key_iv: "d67c9c5bd75e0baa27b9c3d6359223a4"
server_key_version: "encryptionkey@0"
I20260924 01:05:45.660557 23388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:05:45.664482 23403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:45.665956 23388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:05:45.666162 23388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "a9a578d3cbc147e29b5e6f604723805c"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "d109b661c5f7b288c92a85f3a68e1682"
server_key_iv: "d67c9c5bd75e0baa27b9c3d6359223a4"
server_key_version: "encryptionkey@0"
I20260924 01:05:45.666440 23388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:45.682775 23388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:45.686959 23388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:45.687271 23388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:45.688299 23388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:45.689654 23388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:45.689769 23388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:45.689925 23388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:45.689999 23388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:45.723809 23388 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:45101
I20260924 01:05:45.723866 23515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:45101 every 8 connection(s)
I20260924 01:05:45.725129 23388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:05:45.729496 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23388
I20260924 01:05:45.729700 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:05:45.730123 9209 external_mini_cluster.cc:1428] Setting key fb239c4befdd98a2e300afd98ca43ca8
Sep 24 01:05:45 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:45.745744 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:45.727668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:35195 (local address 127.8.254.126:40049)
0924 01:05:45.728115 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:45.728120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:45.729080 (+ 960us) server_negotiation.cc:407] Connection header received
0924 01:05:45.731195 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:45.731200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:45.731303 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:45.731441 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:45.733102 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.733844 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.735534 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.736005 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.736140 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:45.739944 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:45.739949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:45.739953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:45.739987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:45.742593 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:45.743355 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:45.743360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:45.743362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:45.743487 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:45.743940 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:45.743944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:45.743946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:45.744215 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:45.744382 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:45.745109 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 01:05:45.745346 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260924 01:05:45.746825 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:45.728215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:35195)
0924 01:05:45.728918 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:45.728956 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:45.730919 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:45.731703 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:45.731712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:45.732376 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:45.732938 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:45.732955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.734014 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.734018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:45.735362 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:45.735382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:45.736256 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:45.736263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:45.736530 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:45.737300 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:45.737329 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:45.739698 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:45.742735 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:45.742745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:45.742763 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:45.743204 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:45.743602 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:45.743607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:45.743611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:45.743802 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:45.744348 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:45.744356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:45.744718 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 01:05:45.745001 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 01:05:45.745553 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":266,"threads_started":1}
I20260924 01:05:45.748399 23516 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40049
I20260924 01:05:45.748776 23516 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:45.749732 23516 heartbeater.cc:511] Master 127.8.254.126:40049 requested a full tablet report, sending...
I20260924 01:05:45.751588 23193 ts_manager.cc:194] Registered new tserver with Master: a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:45.752842 23193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:35195
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:05:45.785003 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40049
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:46.113659 23522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:46.113950 23522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:46.114044 23522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:46.121301 23522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:46.121403 23522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:46.121522 23522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:05:46.131317 23522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:46.132550 23522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:46.133816 23522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:46.146715 23528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:46.146781 23530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:46.146816 23527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:46.147811 23522 server_base.cc:1034] running on GCE node
I20260924 01:05:46.148376 23522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:46.149290 23522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:46.150519 23522 hybrid_clock.cc:648] HybridClock initialized: now 1790211946150468 us; error 38 us; skew 500 ppm
I20260924 01:05:46.150957 23522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:46 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211946, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:46.154186 23522 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:05:46.155683 23522 webserver.cc:460] Webserver started at http://127.8.254.67:45561/ using document root <none> and password file <none>
I20260924 01:05:46.156458 23522 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:46.156558 23522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:46.156818 23522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:05:46.159809 23522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "859b3ad486c346df91ace7667632a3dd"
format_stamp: "Formatted at 2026-09-24 01:05:46 on dist-test-slave-5c7l"
server_key: "1f8a054715319dfc0061da39b4698da0"
server_key_iv: "b3644a5fb92fb1bd2f16a3fdc4949448"
server_key_version: "encryptionkey@0"
I20260924 01:05:46.160605 23522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "859b3ad486c346df91ace7667632a3dd"
format_stamp: "Formatted at 2026-09-24 01:05:46 on dist-test-slave-5c7l"
server_key: "1f8a054715319dfc0061da39b4698da0"
server_key_iv: "b3644a5fb92fb1bd2f16a3fdc4949448"
server_key_version: "encryptionkey@0"
I20260924 01:05:46.166070 23522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:05:46.169819 23537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:46.171355 23522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:05:46.171538 23522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "859b3ad486c346df91ace7667632a3dd"
format_stamp: "Formatted at 2026-09-24 01:05:46 on dist-test-slave-5c7l"
server_key: "1f8a054715319dfc0061da39b4698da0"
server_key_iv: "b3644a5fb92fb1bd2f16a3fdc4949448"
server_key_version: "encryptionkey@0"
I20260924 01:05:46.171783 23522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:46.205525 23522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:46.209846 23522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:46.210151 23522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:46.211180 23522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:46.212491 23522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:46.212570 23522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:46.212738 23522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:46.212797 23522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:46.233759 23382 heartbeater.cc:503] Master 127.8.254.126:40049 was elected leader, sending a full tablet report...
I20260924 01:05:46.248255 23522 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:33209
I20260924 01:05:46.248291 23649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:33209 every 8 connection(s)
I20260924 01:05:46.249671 23522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:05:46.258049 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23522
I20260924 01:05:46.258282 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:05:46.258740 9209 external_mini_cluster.cc:1428] Setting key 35a02f6d3f1bb7d62a4bf0139e43a78a
Sep 24 01:05:46 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211946, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:46.269721 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.252226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:40733 (local address 127.8.254.126:40049)
0924 01:05:46.252705 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.252710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.253569 (+ 859us) server_negotiation.cc:407] Connection header received
0924 01:05:46.255523 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.255529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.255633 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.255754 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:46.257278 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.258049 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.259876 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.260380 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.260516 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.264081 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:46.264086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:46.264089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:46.264123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:46.266595 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.267323 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.267331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.267339 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.267456 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.267910 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.267913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.267915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.268189 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:46.268364 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.269097 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.269328 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260924 01:05:46.270835 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.252691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:40733)
0924 01:05:46.253399 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:46.253445 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:46.255234 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:46.256004 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:46.256012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:46.256530 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:46.257103 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.257121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.258232 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.258239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.259699 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.259714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.260620 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.260629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.260875 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.261559 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:46.261582 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:46.263862 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:46.266746 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:46.266756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:46.266776 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:46.267187 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:46.267573 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:46.267578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:46.267581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:46.267772 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:46.268352 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:46.268360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:46.268710 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 01:05:46.268983 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 01:05:46.269587 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":272,"threads_started":1}
I20260924 01:05:46.272439 23650 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40049
I20260924 01:05:46.272843 23650 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:46.273772 23650 heartbeater.cc:511] Master 127.8.254.126:40049 requested a full tablet report, sending...
I20260924 01:05:46.275305 23193 ts_manager.cc:194] Registered new tserver with Master: 859b3ad486c346df91ace7667632a3dd (127.8.254.67:33209)
I20260924 01:05:46.276166 23193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:40733
I20260924 01:05:46.286859 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:05:46.304463 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.291461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.8.254.126:40049)
0924 01:05:46.291741 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.291745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.292172 (+ 427us) server_negotiation.cc:407] Connection header received
0924 01:05:46.292287 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.292291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.292356 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.292445 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:46.293844 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.294733 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.296508 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.296944 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.297050 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.299548 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:46.299553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:46.299556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:46.299590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:46.302012 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.302620 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.302624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.302626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.302778 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.303111 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.303115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.303116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.303436 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:46.303669 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.303888 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.304053 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260924 01:05:46.308486 23193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:05:46.315409 23193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:05:46.350226 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.328080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.8.254.65:35361)
0924 01:05:46.339794 (+ 11714us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.339808 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.339863 (+ 55us) server_negotiation.cc:407] Connection header received
0924 01:05:46.340027 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.340042 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.340416 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.341600 (+ 1184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:46.341935 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.344131 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.347338 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.348732 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.348889 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.349148 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.349647 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.349856 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11472,"thread_start_us":11246,"threads_started":1}
I20260924 01:05:46.350286 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.327838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35361 (local address 127.0.0.1:51690)
0924 01:05:46.328594 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:46.328643 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:46.328868 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:46.340799 (+ 11931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:46.340812 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:46.340869 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:46.341307 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.341323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.344357 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.344366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.347141 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.347154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.348971 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.348977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.349320 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.349405 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 01:05:46.349566 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 01:05:46.349771 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":45,"thread_start_us":294,"threads_started":1}
I20260924 01:05:46.353603 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.328894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.8.254.66:45101)
0924 01:05:46.345068 (+ 16174us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.345081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.345116 (+ 35us) server_negotiation.cc:407] Connection header received
0924 01:05:46.345248 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.345258 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.345554 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.345768 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:46.346589 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.348217 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.351391 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.352477 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.352645 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.352856 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.353101 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.353258 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16006,"thread_start_us":15871,"threads_started":1}
I20260924 01:05:46.353660 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.328373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:45101 (local address 127.0.0.1:43118)
0924 01:05:46.342864 (+ 14491us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:46.342891 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:46.343055 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:46.345777 (+ 2722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:46.345782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:46.345805 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:46.346317 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.346335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.348371 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.348377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.351171 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.351187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.352695 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.352702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.352889 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.352923 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:05:46.353034 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 01:05:46.353179 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14302,"mutex_wait_us":144,"thread_start_us":16772,"threads_started":1}
I20260924 01:05:46.354338 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.329059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:33209 (local address 127.0.0.1:45464)
0924 01:05:46.345450 (+ 16391us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:46.345472 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:46.345608 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:46.346243 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:46.346253 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:46.346280 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:46.346823 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.346836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.348660 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.348667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.351087 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.351102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.353230 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.353237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.353402 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.353432 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 01:05:46.353521 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 01:05:46.353677 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16266,"mutex_wait_us":78,"thread_start_us":14098,"threads_started":1}
I20260924 01:05:46.354352 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.329717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.8.254.67:33209)
0924 01:05:46.339293 (+ 9576us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.339307 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.345604 (+ 6297us) server_negotiation.cc:407] Connection header received
0924 01:05:46.345762 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.345769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.346009 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.346228 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:46.347029 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.348520 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.351288 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.352731 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.352872 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.353086 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.353814 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.353977 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9319,"thread_start_us":9116,"threads_started":1}
I20260924 01:05:46.355074 23317 tablet_service.cc:1467] Processing CreateTablet for tablet dd49e33269084740ac80a5f799a20bb6 (DEFAULT_TABLE table=test-table [id=14564d1784ae42a9a7c417746ffcc338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:46.357060 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd49e33269084740ac80a5f799a20bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:46.357980 23585 tablet_service.cc:1467] Processing CreateTablet for tablet dd49e33269084740ac80a5f799a20bb6 (DEFAULT_TABLE table=test-table [id=14564d1784ae42a9a7c417746ffcc338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:46.357981 23451 tablet_service.cc:1467] Processing CreateTablet for tablet dd49e33269084740ac80a5f799a20bb6 (DEFAULT_TABLE table=test-table [id=14564d1784ae42a9a7c417746ffcc338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:46.359403 23451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd49e33269084740ac80a5f799a20bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:46.359401 23585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd49e33269084740ac80a5f799a20bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:46.368994 23669 tablet_bootstrap.cc:492] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Bootstrap starting.
I20260924 01:05:46.373049 23669 tablet_bootstrap.cc:654] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:46.374478 23669 log.cc:824] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:46.375473 23671 tablet_bootstrap.cc:492] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Bootstrap starting.
I20260924 01:05:46.376307 23670 tablet_bootstrap.cc:492] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Bootstrap starting.
I20260924 01:05:46.378482 23669 tablet_bootstrap.cc:492] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: No bootstrap required, opened a new log
I20260924 01:05:46.378764 23669 ts_tablet_manager.cc:1397] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260924 01:05:46.379918 23670 tablet_bootstrap.cc:654] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:46.379921 23671 tablet_bootstrap.cc:654] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:46.381201 23670 log.cc:824] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:46.381201 23671 log.cc:824] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:46.384114 23671 tablet_bootstrap.cc:492] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: No bootstrap required, opened a new log
I20260924 01:05:46.384385 23671 ts_tablet_manager.cc:1397] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 01:05:46.384960 23670 tablet_bootstrap.cc:492] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: No bootstrap required, opened a new log
I20260924 01:05:46.385226 23670 ts_tablet_manager.cc:1397] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 01:05:46.388104 23669 raft_consensus.cc:348] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.388523 23669 raft_consensus.cc:374] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:46.388622 23669 raft_consensus.cc:729] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6d3a1f06cf435daa8e6a341c4d1dad, State: Initialized, Role: FOLLOWER
I20260924 01:05:46.389272 23669 consensus_queue.cc:260] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.391142 23669 ts_tablet_manager.cc:1428] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 01:05:46.393235 23671 raft_consensus.cc:348] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.393718 23671 raft_consensus.cc:374] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:46.393824 23671 raft_consensus.cc:729] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859b3ad486c346df91ace7667632a3dd, State: Initialized, Role: FOLLOWER
I20260924 01:05:46.394168 23670 raft_consensus.cc:348] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.394624 23670 raft_consensus.cc:374] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:46.394723 23670 raft_consensus.cc:729] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a578d3cbc147e29b5e6f604723805c, State: Initialized, Role: FOLLOWER
I20260924 01:05:46.394582 23671 consensus_queue.cc:260] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.395291 23670 consensus_queue.cc:260] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.396126 23650 heartbeater.cc:503] Master 127.8.254.126:40049 was elected leader, sending a full tablet report...
I20260924 01:05:46.396735 23671 ts_tablet_manager.cc:1428] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260924 01:05:46.397753 23516 heartbeater.cc:503] Master 127.8.254.126:40049 was elected leader, sending a full tablet report...
I20260924 01:05:46.398368 23670 ts_tablet_manager.cc:1428] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
W20260924 01:05:46.452908 23383 tablet.cc:2369] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:05:46.477200 23517 tablet.cc:2369] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:05:46.501299 23651 tablet.cc:2369] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:46.599371 23676 raft_consensus.cc:482] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:05:46.599752 23676 raft_consensus.cc:504] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.601452 23676 leader_election.cc:288] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361), a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:46.607216 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.602026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35361 (local address 127.8.254.67:54659)
0924 01:05:46.602229 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:46.602250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:46.602408 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:46.602756 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:46.602760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:46.602791 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:46.602977 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.602986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.604035 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.604041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.605463 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.605476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.606692 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.606697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.606833 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.606858 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 01:05:46.606933 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 01:05:46.607024 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 01:05:46.607380 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.601896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:54659 (local address 127.8.254.65:35361)
0924 01:05:46.602114 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.602121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.602341 (+ 220us) server_negotiation.cc:407] Connection header received
0924 01:05:46.602555 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.602565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.602624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.602722 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:46.603118 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.603887 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.605618 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.606538 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.606701 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.606773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.606990 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.607121 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:05:46.608536 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.602030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:45101 (local address 127.8.254.67:51561)
0924 01:05:46.602985 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:46.603000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:46.603152 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:46.603470 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:46.603474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:46.603499 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:46.603816 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.603826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.604888 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.604893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.606354 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:46.606367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.607961 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.607969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:46.608107 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.608129 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 01:05:46.608208 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 01:05:46.608287 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":124,"thread_start_us":411,"threads_started":1}
I20260924 01:05:46.608669 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.601943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:51561 (local address 127.8.254.66:45101)
0924 01:05:46.602127 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.602135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.603088 (+ 953us) server_negotiation.cc:407] Connection header received
0924 01:05:46.603287 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.603296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.603357 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.603455 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:46.603998 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.604738 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.606526 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.607760 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.607887 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.607936 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.608278 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.608410 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:05:46.608484 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd49e33269084740ac80a5f799a20bb6" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" is_pre_election: true
I20260924 01:05:46.609123 23337 raft_consensus.cc:2454] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 0.
I20260924 01:05:46.609583 23471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd49e33269084740ac80a5f799a20bb6" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a578d3cbc147e29b5e6f604723805c" is_pre_election: true
I20260924 01:05:46.609916 23541 leader_election.cc:302] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859b3ad486c346df91ace7667632a3dd, db6d3a1f06cf435daa8e6a341c4d1dad; no voters:
I20260924 01:05:46.610139 23471 raft_consensus.cc:2454] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 0.
I20260924 01:05:46.610335 23676 raft_consensus.cc:2781] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:05:46.610504 23676 raft_consensus.cc:482] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:05:46.610605 23676 raft_consensus.cc:3037] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:46.612802 23676 raft_consensus.cc:504] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.613392 23676 leader_election.cc:288] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 election: Requested vote from peers db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361), a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:46.613837 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd49e33269084740ac80a5f799a20bb6" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
I20260924 01:05:46.613951 23471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd49e33269084740ac80a5f799a20bb6" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a578d3cbc147e29b5e6f604723805c"
I20260924 01:05:46.614063 23337 raft_consensus.cc:3037] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:46.614198 23471 raft_consensus.cc:3037] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:46.616312 23337 raft_consensus.cc:2454] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 1.
I20260924 01:05:46.616410 23471 raft_consensus.cc:2454] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 1.
I20260924 01:05:46.616734 23541 leader_election.cc:302] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859b3ad486c346df91ace7667632a3dd, db6d3a1f06cf435daa8e6a341c4d1dad; no voters:
I20260924 01:05:46.617014 23676 raft_consensus.cc:2781] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:46.617434 23676 raft_consensus.cc:686] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 1 LEADER]: Becoming Leader. State: Replica: 859b3ad486c346df91ace7667632a3dd, State: Running, Role: LEADER
I20260924 01:05:46.617985 23676 consensus_queue.cc:237] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:46.622694 23193 catalog_manager.cc:5495] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 859b3ad486c346df91ace7667632a3dd (127.8.254.67). New cstate: current_term: 1 leader_uuid: "859b3ad486c346df91ace7667632a3dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } health_report { overall_health: UNKNOWN } } }
I20260924 01:05:46.650926 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.645492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.8.254.67:33209)
0924 01:05:46.645736 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.645742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.645762 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:05:46.645959 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.645963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.646040 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.646198 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:05:46.646646 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.647366 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.648784 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.649184 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.649323 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.649812 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:05:46.650340 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:05:46.650512 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.650565 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.650647 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:05:46.659215 23471 raft_consensus.cc:1261] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Refusing update from remote peer 859b3ad486c346df91ace7667632a3dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:46.659341 23337 raft_consensus.cc:1261] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Refusing update from remote peer 859b3ad486c346df91ace7667632a3dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:46.660250 23679 consensus_queue.cc:1035] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:46.660655 23676 consensus_queue.cc:1035] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:46.675180 23682 mvcc.cc:205] Tried to move back new op lower bound from 7332708133506240512 to 7332708133351444480. Current Snapshot: MvccSnapshot[applied={T|T < 7332708133506240512}]
I20260924 01:05:46.690388 23192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:05:46.690753 23192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:05:46.695006 23192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Sending DeleteTablet for 3 replicas of tablet dd49e33269084740ac80a5f799a20bb6
I20260924 01:05:46.696514 23451 tablet_service.cc:1514] Processing DeleteTablet for tablet dd49e33269084740ac80a5f799a20bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:46 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:43118
I20260924 01:05:46.696585 23317 tablet_service.cc:1514] Processing DeleteTablet for tablet dd49e33269084740ac80a5f799a20bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:46 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:51690
I20260924 01:05:46.696794 23585 tablet_service.cc:1514] Processing DeleteTablet for tablet dd49e33269084740ac80a5f799a20bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:46 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:45464
I20260924 01:05:46.697670 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23254
I20260924 01:05:46.707706 23695 tablet_replica.cc:330] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: stopping tablet replica
I20260924 01:05:46.709192 23695 raft_consensus.cc:2229] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:05:46.710873 23695 raft_consensus.cc:2258] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:46.711869 23694 tablet_replica.cc:330] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: stopping tablet replica
I20260924 01:05:46.712309 23694 raft_consensus.cc:2229] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:05:46.712839 23694 raft_consensus.cc:2258] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:46.714811 23694 ts_tablet_manager.cc:1905] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:46.714820 23695 ts_tablet_manager.cc:1905] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:46.717013 23693 tablet_replica.cc:330] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: stopping tablet replica
I20260924 01:05:46.717450 23693 raft_consensus.cc:2229] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:05:46.717963 23693 raft_consensus.cc:2258] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:46.722426 23694 ts_tablet_manager.cc:1918] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:46.722604 23694 log.cc:1196] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/dd49e33269084740ac80a5f799a20bb6
I20260924 01:05:46.723169 23694 ts_tablet_manager.cc:1939] T dd49e33269084740ac80a5f799a20bb6 P db6d3a1f06cf435daa8e6a341c4d1dad: Deleting consensus metadata
I20260924 01:05:46.723745 23693 ts_tablet_manager.cc:1905] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:46.724814 23695 ts_tablet_manager.cc:1918] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:46.725042 23695 log.cc:1196] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/dd49e33269084740ac80a5f799a20bb6
I20260924 01:05:46.725101 23180 catalog_manager.cc:4841] TS db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361): tablet dd49e33269084740ac80a5f799a20bb6 (table test-table [id=14564d1784ae42a9a7c417746ffcc338]) successfully deleted
I20260924 01:05:46.725677 23695 ts_tablet_manager.cc:1939] T dd49e33269084740ac80a5f799a20bb6 P 859b3ad486c346df91ace7667632a3dd: Deleting consensus metadata
I20260924 01:05:46.727348 23180 catalog_manager.cc:4841] TS 859b3ad486c346df91ace7667632a3dd (127.8.254.67:33209): tablet dd49e33269084740ac80a5f799a20bb6 (table test-table [id=14564d1784ae42a9a7c417746ffcc338]) successfully deleted
I20260924 01:05:46.730499 23693 ts_tablet_manager.cc:1918] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:46.730664 23693 log.cc:1196] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/dd49e33269084740ac80a5f799a20bb6
I20260924 01:05:46.731213 23693 ts_tablet_manager.cc:1939] T dd49e33269084740ac80a5f799a20bb6 P a9a578d3cbc147e29b5e6f604723805c: Deleting consensus metadata
Sep 24 01:05:46 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:05:46.733055 23179 catalog_manager.cc:4841] TS a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101): tablet dd49e33269084740ac80a5f799a20bb6 (table test-table [id=14564d1784ae42a9a7c417746ffcc338]) successfully deleted
I20260924 01:05:46.741571 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.699112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.8.254.65:35361)
0924 01:05:46.699339 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.699347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.713358 (+ 14011us) server_negotiation.cc:407] Connection header received
0924 01:05:46.713944 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.713950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.714046 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.714120 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:46.718381 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.719550 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.727137 (+ 7587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.727802 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.727966 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.734011 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:46.734029 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:46.734048 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:46.734095 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:46.737986 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.739007 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.739019 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.739022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.739181 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.739629 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.739634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.739638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.740071 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:46.740375 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.740946 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.741185 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:05:46.743427 23377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:46.838093 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23254
I20260924 01:05:46.863291 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23388
Sep 24 01:05:46 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:05:46.884974 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:46.864135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.8.254.66:45101)
0924 01:05:46.864318 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:46.864324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:46.864348 (+ 24us) server_negotiation.cc:407] Connection header received
0924 01:05:46.864575 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:46.864581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:46.864662 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:46.864781 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:46.866042 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.866867 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.868231 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:46.868629 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:46.868761 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:46.879037 (+ 10276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:46.879055 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:46.879072 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:46.879113 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:46.881983 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.882861 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.882868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.882870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.882991 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:46.883444 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:46.883451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:46.883457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:46.883939 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:46.884159 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:46.884501 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 01:05:46.884664 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 01:05:46.885648 23511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:46.979079 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23388
I20260924 01:05:47.003753 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23522
Sep 24 01:05:47 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:05:47.020226 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:47.004644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.8.254.67:33209)
0924 01:05:47.004849 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:47.004857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:47.004968 (+ 111us) server_negotiation.cc:407] Connection header received
0924 01:05:47.005200 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:47.005205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:47.005299 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:47.005419 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:47.006801 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:47.007720 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:47.009014 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:47.009474 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:47.009626 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:47.013989 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:47.014007 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:47.014023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:47.014071 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:47.017227 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:47.017980 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:47.017994 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:47.017997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:47.018139 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:47.018573 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:47.018579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:47.018582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:47.018992 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:47.019318 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:47.019619 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 01:05:47.019856 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 01:05:47.021004 23645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:47.145596 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23522
I20260924 01:05:47.166549 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23161
I20260924 01:05:47.167711 23223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:47.281225 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23161
I20260924 01:05:47.305152 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40049
--webserver_interface=127.8.254.126
--webserver_port=37663
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:47.640949 23701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:47.641320 23701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:47.641484 23701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:47.648927 23701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:47.649082 23701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:47.649205 23701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:47.649304 23701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:47.649390 23701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:47.659550 23701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:40049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=37663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:47.660964 23701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:47.662349 23701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:47.687752 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:47.687842 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:47.687783 23707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:47.688819 23701 server_base.cc:1034] running on GCE node
I20260924 01:05:47.689821 23701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:47.691471 23701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:47.692720 23701 hybrid_clock.cc:648] HybridClock initialized: now 1790211947692673 us; error 53 us; skew 500 ppm
I20260924 01:05:47.693140 23701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:47 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211947, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:47.696923 23701 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:47.699047 23701 webserver.cc:460] Webserver started at http://127.8.254.126:37663/ using document root <none> and password file <none>
I20260924 01:05:47.699944 23701 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:47.700133 23701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:47.706888 23701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 01:05:47.710163 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:47.711925 23701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:05:47.712239 23701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "76f1762dc09442f4a565fe4410eebc46"
format_stamp: "Formatted at 2026-09-24 01:05:44 on dist-test-slave-5c7l"
server_key: "7d9921ebedd97e941df0fe6bbf29fc9f"
server_key_iv: "4b57c40e95303d115aacf7a724f033c9"
server_key_version: "encryptionkey@0"
I20260924 01:05:47.713486 23701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:47.739064 23701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:47.743670 23701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:47.743986 23701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:47.764837 23701 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:40049
I20260924 01:05:47.764860 23768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:40049 every 8 connection(s)
I20260924 01:05:47.766649 23701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:47.769443 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23701
I20260924 01:05:47.770494 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:35361
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:40049
--webserver_port=43379
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:47.776748 23769 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:05:47.781277 23769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap starting.
I20260924 01:05:47.801826 23769 log.cc:824] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:47.814374 23769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:05:47.814937 23769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap complete.
I20260924 01:05:47.824613 23769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:47.825464 23769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f1762dc09442f4a565fe4410eebc46, State: Initialized, Role: FOLLOWER
I20260924 01:05:47.826241 23769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:47.826510 23769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:05:47.826634 23769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:05:47.826826 23769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 1 FOLLOWER]: Advancing to term 2
I20260924 01:05:47.830819 23769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:47.831375 23769 leader_election.cc:302] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f1762dc09442f4a565fe4410eebc46; no voters:
I20260924 01:05:47.832026 23769 leader_election.cc:288] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 01:05:47.832172 23773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 01:05:47.833510 23773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 LEADER]: Becoming Leader. State: Replica: 76f1762dc09442f4a565fe4410eebc46, State: Running, Role: LEADER
I20260924 01:05:47.834177 23773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:47.834534 23769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:47.837224 23774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76f1762dc09442f4a565fe4410eebc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } } }
I20260924 01:05:47.837772 23774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:47.837446 23775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f1762dc09442f4a565fe4410eebc46. Latest consensus state: current_term: 2 leader_uuid: "76f1762dc09442f4a565fe4410eebc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } } }
I20260924 01:05:47.838608 23775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:47.849628 23780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:47.860105 23780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:05:47.862139 23780 catalog_manager.cc:1259] Loaded cluster ID: 00c7990f408246f8af76f10c44345303
I20260924 01:05:47.862411 23780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:05:47.866663 23780 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 01:05:47.893693 23780 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 01:05:47.906418 23780 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:05:47.908797 23780 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Loaded TSK: 0
I20260924 01:05:47.910983 23780 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 01:05:48.153602 23771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:48.153896 23771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:48.154050 23771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:48.161598 23771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:48.161720 23771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:48.161895 23771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:05:48.173421 23771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=43379
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:48.174887 23771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:48.176205 23771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:48.189100 23796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:48.189102 23797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:48.190361 23799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:48.190714 23771 server_base.cc:1034] running on GCE node
I20260924 01:05:48.191341 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:48.192263 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:48.193543 23771 hybrid_clock.cc:648] HybridClock initialized: now 1790211948193448 us; error 56 us; skew 500 ppm
I20260924 01:05:48.193951 23771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:48 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:48.198097 23771 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:05:48.199817 23771 webserver.cc:460] Webserver started at http://127.8.254.65:43379/ using document root <none> and password file <none>
I20260924 01:05:48.200543 23771 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:48.200696 23771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:48.206046 23771 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:05:48.208968 23806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:48.210589 23771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:05:48.210777 23771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "cd823dd8f79cbeac965db70a6f454b12"
server_key_iv: "b49aa0d682d2112b471b199e3de55dba"
server_key_version: "encryptionkey@0"
I20260924 01:05:48.211783 23771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:48.243052 23771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:48.247591 23771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:48.247891 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:48.248988 23771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:48.250540 23771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:48.250671 23771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:48.250800 23771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:48.250888 23771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:48.285006 23771 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:35361
I20260924 01:05:48.285097 23918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:35361 every 8 connection(s)
I20260924 01:05:48.286554 23771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:05:48.288587 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23771
I20260924 01:05:48.289391 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:45101
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:40049
--webserver_port=34359
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:05:48 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:48.313184 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:48.289387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:59367 (local address 127.8.254.126:40049)
0924 01:05:48.290060 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:48.290076 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:48.290816 (+ 740us) server_negotiation.cc:407] Connection header received
0924 01:05:48.293030 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:48.293044 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:48.293418 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:48.293903 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:48.295533 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.296801 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.299445 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.300099 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.300254 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:48.304867 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:48.304875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:48.304891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:48.304928 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:48.308253 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:48.309319 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:48.309326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:48.309331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:48.309492 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:48.310084 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:48.310088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:48.310090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:48.310704 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:48.310869 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:48.311911 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0924 01:05:48.312247 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
I20260924 01:05:48.313840 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:48.289742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:59367)
0924 01:05:48.290620 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:48.290665 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:48.292436 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:48.294045 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:48.294058 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:48.294725 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:48.295285 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:48.295304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.296984 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.296993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:48.299263 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:48.299281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.300345 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.300354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:48.300690 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:48.301428 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:48.301464 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:48.304543 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:48.308419 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:48.308430 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:48.308451 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:48.309160 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:48.309614 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:48.309622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:48.309629 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:48.309917 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:48.310862 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:48.310873 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:48.311346 (+ 473us) client_negotiation.cc:769] Sending connection context
0924 01:05:48.311704 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 01:05:48.312344 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":414,"threads_started":1}
I20260924 01:05:48.316807 23919 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40049
I20260924 01:05:48.317133 23919 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:48.318022 23919 heartbeater.cc:511] Master 127.8.254.126:40049 requested a full tablet report, sending...
I20260924 01:05:48.320497 23734 ts_manager.cc:194] Registered new tserver with Master: db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361)
I20260924 01:05:48.322678 23734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:59367
W20260924 01:05:48.626693 23923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:48.626976 23923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:48.627074 23923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:48.634560 23923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:48.634677 23923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:48.634792 23923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:05:48.644668 23923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:45101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=34359
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:48.645963 23923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:48.647418 23923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:48.661188 23928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:48.661144 23931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:48.661248 23929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:48.662300 23923 server_base.cc:1034] running on GCE node
I20260924 01:05:48.663008 23923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:48.664090 23923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:48.665326 23923 hybrid_clock.cc:648] HybridClock initialized: now 1790211948665251 us; error 61 us; skew 500 ppm
I20260924 01:05:48.665663 23923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:48 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:48.668848 23923 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:05:48.670337 23923 webserver.cc:460] Webserver started at http://127.8.254.66:34359/ using document root <none> and password file <none>
I20260924 01:05:48.671016 23923 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:48.671125 23923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:48.676806 23923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 01:05:48.679553 23938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:48.680974 23923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:05:48.681160 23923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "a9a578d3cbc147e29b5e6f604723805c"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "d109b661c5f7b288c92a85f3a68e1682"
server_key_iv: "d67c9c5bd75e0baa27b9c3d6359223a4"
server_key_version: "encryptionkey@0"
I20260924 01:05:48.682186 23923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:48.707930 23923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:48.712095 23923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:48.712409 23923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:48.713431 23923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:48.714886 23923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:48.715009 23923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:48.715142 23923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:48.715240 23923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:48.748715 23923 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:45101
I20260924 01:05:48.748854 24050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:45101 every 8 connection(s)
I20260924 01:05:48.750412 23923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:05:48.755674 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 23923
I20260924 01:05:48.756309 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:33209
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:40049
--webserver_port=45561
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:05:48 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:48.772853 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:48.753052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:41255 (local address 127.8.254.126:40049)
0924 01:05:48.753234 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:48.753239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:48.754439 (+ 1200us) server_negotiation.cc:407] Connection header received
0924 01:05:48.756419 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:48.756425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:48.756524 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:48.756636 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:48.758151 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.758910 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.760499 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.760918 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.761052 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:48.765142 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:48.765148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:48.765153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:48.765193 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:48.768899 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:48.769954 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:48.769960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:48.769964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:48.770106 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:48.770655 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:48.770660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:48.770663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:48.771058 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:48.771202 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:48.772181 (+ 979us) server_negotiation.cc:299] Negotiation successful
0924 01:05:48.772502 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:05:48.774469 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:48.753528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:41255)
0924 01:05:48.754292 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:48.754333 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:48.756138 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:48.756894 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:48.756903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:48.757402 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:48.757974 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:48.757992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.759081 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.759085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:48.760297 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:48.760310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:48.761158 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:48.761165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:48.761401 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:48.762091 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:48.762113 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:48.764907 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:48.769084 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:48.769096 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:48.769116 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:48.769817 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:48.770251 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:48.770259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:48.770266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:48.770521 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:48.771220 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:48.771231 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:48.771717 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 01:05:48.772035 (+ 318us) client_negotiation.cc:240] Negotiation successful
0924 01:05:48.772678 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260924 01:05:48.776368 24051 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40049
I20260924 01:05:48.776760 24051 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:48.777742 24051 heartbeater.cc:511] Master 127.8.254.126:40049 requested a full tablet report, sending...
I20260924 01:05:48.779630 23734 ts_manager.cc:194] Registered new tserver with Master: a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:48.780755 23734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:41255
W20260924 01:05:49.086470 24054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:49.086805 24054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:49.086969 24054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:49.094374 24054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:49.094520 24054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:49.094699 24054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:05:49.104192 24054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:33209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=45561
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:49.105487 24054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:49.106892 24054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:49.120344 24059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:49.120378 24060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:49.120298 24062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:49.121080 24054 server_base.cc:1034] running on GCE node
I20260924 01:05:49.121708 24054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:49.122740 24054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:49.123939 24054 hybrid_clock.cc:648] HybridClock initialized: now 1790211949123881 us; error 59 us; skew 500 ppm
I20260924 01:05:49.124275 24054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:49 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:49.127583 24054 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:05:49.129094 24054 webserver.cc:460] Webserver started at http://127.8.254.67:45561/ using document root <none> and password file <none>
I20260924 01:05:49.129807 24054 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:49.129953 24054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:49.135474 24054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 01:05:49.138332 24069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:49.139663 24054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:05:49.139851 24054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "859b3ad486c346df91ace7667632a3dd"
format_stamp: "Formatted at 2026-09-24 01:05:46 on dist-test-slave-5c7l"
server_key: "1f8a054715319dfc0061da39b4698da0"
server_key_iv: "b3644a5fb92fb1bd2f16a3fdc4949448"
server_key_version: "encryptionkey@0"
I20260924 01:05:49.140862 24054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:49.160778 24054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:49.165359 24054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:49.165666 24054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:49.166738 24054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:49.168171 24054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:49.168318 24054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:49.168483 24054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:49.168578 24054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:49.205714 24054 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:33209
I20260924 01:05:49.205818 24181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:33209 every 8 connection(s)
I20260924 01:05:49.207278 24054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:05:49.210839 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24054
Sep 24 01:05:49 dist-test-slave-5c7l krb5kdc[23148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:05:49.230242 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.211878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.8.254.126:40049)
0924 01:05:49.212347 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.212351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.214700 (+ 2349us) server_negotiation.cc:407] Connection header received
0924 01:05:49.214907 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.214912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.214997 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.215126 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:49.216723 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.217771 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.219399 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.219870 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.220004 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.223166 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:49.223171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:49.223175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:49.223206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:49.226918 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:49.227842 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:49.227846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:49.227848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:49.227962 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:49.228397 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:49.228404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:49.228408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:49.228773 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:49.229079 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.229461 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.229744 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260924 01:05:49.231993 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.209947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:51159 (local address 127.8.254.126:40049)
0924 01:05:49.210140 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.210145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.211532 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 01:05:49.214955 (+ 3423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.214961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.215044 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.215157 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:49.216950 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.217932 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.219707 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.220240 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.220385 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.224540 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:49.224544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:49.224548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:49.224577 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:49.228162 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:49.229170 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:49.229175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:49.229178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:49.229327 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:49.230041 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:49.230045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:49.230047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:49.230405 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:49.230543 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.231444 (+ 901us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.231643 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:05:49.233057 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.210452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:51159)
0924 01:05:49.211338 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.211397 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.214662 (+ 3265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.215481 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.215489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.216070 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:05:49.216770 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.216789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.218122 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.218127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.219552 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.219565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.220514 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.220523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.220777 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.221376 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:05:49.221404 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:05:49.224313 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:05:49.228297 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:49.228305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:49.228343 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:49.229008 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:49.229525 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:05:49.229531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:05:49.229534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:05:49.229858 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:05:49.230532 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:05:49.230541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:05:49.231015 (+ 474us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.231347 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.231909 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":345,"threads_started":1}
I20260924 01:05:49.234408 24182 heartbeater.cc:347] Connected to a master server at 127.8.254.126:40049
I20260924 01:05:49.234748 24182 heartbeater.cc:464] Registering TS with master...
I20260924 01:05:49.235503 24182 heartbeater.cc:511] Master 127.8.254.126:40049 requested a full tablet report, sending...
I20260924 01:05:49.236876 23734 ts_manager.cc:194] Registered new tserver with Master: 859b3ad486c346df91ace7667632a3dd (127.8.254.67:33209)
I20260924 01:05:49.237713 23734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:51159
I20260924 01:05:49.245316 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:05:49.252698 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.246309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.8.254.126:40049)
0924 01:05:49.246472 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.246477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.247018 (+ 541us) server_negotiation.cc:407] Connection header received
0924 01:05:49.247204 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.247209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.247356 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.247488 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:05:49.247949 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.248674 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.250423 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.250985 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.251092 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.251578 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:05:49.252217 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:05:49.252334 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.252435 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.252503 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:05:49.253674 23734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:05:49.260972 23734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:05:49.287104 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.272996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35361 (local address 127.0.0.1:51712)
0924 01:05:49.273830 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.273891 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.274082 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.275749 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.275759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.275812 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.276335 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.276353 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.279050 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.279060 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.282529 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.282555 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.285689 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.285699 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.285993 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.286069 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.286276 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.286565 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":383,"threads_started":1}
I20260924 01:05:49.289149 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.273278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.8.254.65:35361)
0924 01:05:49.274655 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.274667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.274705 (+ 38us) server_negotiation.cc:407] Connection header received
0924 01:05:49.274836 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.274848 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.275165 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.275383 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.276626 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.278825 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.282786 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.285002 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.285186 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.285487 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.288481 (+ 2994us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.288672 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":1055,"threads_started":1}
I20260924 01:05:49.296880 23854 tablet_service.cc:1467] Processing CreateTablet for tablet 1a471689c7fe4a798721a2e2bd0836d0 (DEFAULT_TABLE table=test-table [id=a36eaddab02b48ca9fe4eff6dddd5b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:49.298568 23854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a471689c7fe4a798721a2e2bd0836d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:49.299517 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.274291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:33209 (local address 127.0.0.1:45480)
0924 01:05:49.288259 (+ 13968us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.288275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.288363 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.292824 (+ 4461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.292832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.292860 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.293031 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.293041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.295364 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.295369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.297072 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.297086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.298481 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.298487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.298766 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.298800 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.298890 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.299228 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13847,"thread_start_us":17294,"threads_started":1}
I20260924 01:05:49.299535 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.274918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.8.254.67:33209)
0924 01:05:49.290352 (+ 15434us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.290365 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.290415 (+ 50us) server_negotiation.cc:407] Connection header received
0924 01:05:49.290550 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.290562 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.290887 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.291164 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.293242 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.294961 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.297244 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.298317 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.298459 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.298948 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.299039 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.299222 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15180,"thread_start_us":15031,"threads_started":1}
I20260924 01:05:49.300601 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.273738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:45101 (local address 127.0.0.1:43128)
0924 01:05:49.288090 (+ 14352us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.288119 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.288292 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.293232 (+ 4940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.293235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.293258 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.293608 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.293622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.295743 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.295750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.297807 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.297822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.299718 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.299725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.299921 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.299959 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.300036 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.300184 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14211,"mutex_wait_us":1,"thread_start_us":14038,"threads_started":1}
I20260924 01:05:49.301318 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.274107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.8.254.66:45101)
0924 01:05:49.292216 (+ 18109us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.292225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.292262 (+ 37us) server_negotiation.cc:407] Connection header received
0924 01:05:49.292358 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.292366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.292618 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.293801 (+ 1183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.294044 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.295567 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.298028 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.299465 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.300609 (+ 1144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.300838 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.300910 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.301109 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17929,"thread_start_us":17803,"threads_started":1}
I20260924 01:05:49.304502 24117 tablet_service.cc:1467] Processing CreateTablet for tablet 1a471689c7fe4a798721a2e2bd0836d0 (DEFAULT_TABLE table=test-table [id=a36eaddab02b48ca9fe4eff6dddd5b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:49.305060 23986 tablet_service.cc:1467] Processing CreateTablet for tablet 1a471689c7fe4a798721a2e2bd0836d0 (DEFAULT_TABLE table=test-table [id=a36eaddab02b48ca9fe4eff6dddd5b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:05:49.305989 24117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a471689c7fe4a798721a2e2bd0836d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:49.306367 23986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a471689c7fe4a798721a2e2bd0836d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:05:49.312213 24198 tablet_bootstrap.cc:492] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Bootstrap starting.
I20260924 01:05:49.316169 24198 tablet_bootstrap.cc:654] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:49.317334 24198 log.cc:824] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:49.323189 24201 tablet_bootstrap.cc:492] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Bootstrap starting.
I20260924 01:05:49.324368 24200 tablet_bootstrap.cc:492] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Bootstrap starting.
I20260924 01:05:49.325906 23919 heartbeater.cc:503] Master 127.8.254.126:40049 was elected leader, sending a full tablet report...
I20260924 01:05:49.326512 24198 tablet_bootstrap.cc:492] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: No bootstrap required, opened a new log
I20260924 01:05:49.326793 24198 ts_tablet_manager.cc:1397] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260924 01:05:49.327169 24201 tablet_bootstrap.cc:654] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:49.328464 24201 log.cc:824] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:49.329968 24200 tablet_bootstrap.cc:654] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Neither blocks nor log segments found. Creating new log.
I20260924 01:05:49.331331 24200 log.cc:824] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:49.332708 24201 tablet_bootstrap.cc:492] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: No bootstrap required, opened a new log
I20260924 01:05:49.333068 24201 ts_tablet_manager.cc:1397] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 01:05:49.335783 24198 raft_consensus.cc:348] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.336180 24198 raft_consensus.cc:374] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:49.336301 24198 raft_consensus.cc:729] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6d3a1f06cf435daa8e6a341c4d1dad, State: Initialized, Role: FOLLOWER
I20260924 01:05:49.336995 24200 tablet_bootstrap.cc:492] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: No bootstrap required, opened a new log
I20260924 01:05:49.336941 24198 consensus_queue.cc:260] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.337301 24200 ts_tablet_manager.cc:1397] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 01:05:49.338881 24198 ts_tablet_manager.cc:1428] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260924 01:05:49.345216 24201 raft_consensus.cc:348] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.345716 24201 raft_consensus.cc:374] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:49.345822 24201 raft_consensus.cc:729] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859b3ad486c346df91ace7667632a3dd, State: Initialized, Role: FOLLOWER
I20260924 01:05:49.346505 24201 consensus_queue.cc:260] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.347589 24182 heartbeater.cc:503] Master 127.8.254.126:40049 was elected leader, sending a full tablet report...
I20260924 01:05:49.348006 24200 raft_consensus.cc:348] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.348404 24201 ts_tablet_manager.cc:1428] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 01:05:49.348484 24200 raft_consensus.cc:374] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:05:49.348584 24200 raft_consensus.cc:729] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a578d3cbc147e29b5e6f604723805c, State: Initialized, Role: FOLLOWER
I20260924 01:05:49.349216 24200 consensus_queue.cc:260] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.350366 24051 heartbeater.cc:503] Master 127.8.254.126:40049 was elected leader, sending a full tablet report...
I20260924 01:05:49.351208 24200 ts_tablet_manager.cc:1428] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260924 01:05:49.458917 24183 tablet.cc:2369] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:05:49.502521 24052 tablet.cc:2369] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:05:49.539021 23920 tablet.cc:2369] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:05:49.750967 24205 raft_consensus.cc:482] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:05:49.751408 24205 raft_consensus.cc:504] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.753290 24205 leader_election.cc:288] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361), a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:49.757004 24204 raft_consensus.cc:482] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:05:49.757311 24204 raft_consensus.cc:504] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.759496 24204 leader_election.cc:288] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859b3ad486c346df91ace7667632a3dd (127.8.254.67:33209), a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:49.762593 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.753463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:53995 (local address 127.8.254.65:35361)
0924 01:05:49.753690 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.753697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.754662 (+ 965us) server_negotiation.cc:407] Connection header received
0924 01:05:49.754982 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.754988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.755066 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.755152 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.756017 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.757177 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.759996 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.761294 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.761421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.761495 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.762160 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.762325 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:05:49.762616 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.754408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:52761 (local address 127.8.254.66:45101)
0924 01:05:49.754582 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.754588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.755092 (+ 504us) server_negotiation.cc:407] Connection header received
0924 01:05:49.755337 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.755342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.755412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.755510 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.756288 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.757428 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.759971 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.761302 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.761421 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.761495 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.762160 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.762331 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 01:05:49.763491 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.754425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:45101 (local address 127.8.254.67:52761)
0924 01:05:49.754998 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.755014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.755182 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.755541 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.755546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.755575 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.756119 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.756130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.757604 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.757609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.759785 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.759793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.761696 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.761700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.761959 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.761993 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.762857 (+ 864us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.762997 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":80,"thread_start_us":318,"threads_started":1}
I20260924 01:05:49.763512 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.753758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:35361 (local address 127.8.254.67:53995)
0924 01:05:49.754552 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.754572 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.754783 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.755360 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.755364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.755389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.755825 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.755839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.757372 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.757377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.759776 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.759793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.761645 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.761651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.761945 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.761982 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.762827 (+ 845us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.762996 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":103,"thread_start_us":852,"threads_started":1}
I20260924 01:05:49.764724 23874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a471689c7fe4a798721a2e2bd0836d0" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" is_pre_election: true
I20260924 01:05:49.765341 23874 raft_consensus.cc:2454] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 0.
I20260924 01:05:49.766340 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a471689c7fe4a798721a2e2bd0836d0" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a578d3cbc147e29b5e6f604723805c" is_pre_election: true
I20260924 01:05:49.767066 24006 raft_consensus.cc:2454] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 0.
I20260924 01:05:49.768334 24072 leader_election.cc:302] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859b3ad486c346df91ace7667632a3dd, a9a578d3cbc147e29b5e6f604723805c; no voters:
I20260924 01:05:49.768988 24205 raft_consensus.cc:2781] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:05:49.769232 24205 raft_consensus.cc:482] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:05:49.769418 24205 raft_consensus.cc:3037] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:49.773823 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.760489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:34347 (local address 127.8.254.66:45101)
0924 01:05:49.762821 (+ 2332us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.762826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.766226 (+ 3400us) server_negotiation.cc:407] Connection header received
0924 01:05:49.766428 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.766433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.766537 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.766623 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.767336 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.768406 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.771348 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.772561 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.772707 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.772789 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.773352 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.773484 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2227,"thread_start_us":4813,"threads_started":1}
I20260924 01:05:49.774549 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.760056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:45101 (local address 127.8.254.65:34347)
0924 01:05:49.766085 (+ 6029us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.766105 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.766275 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.766777 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.766782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.766808 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.767188 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.767199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.768591 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.768596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.771170 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.771186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.772991 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.772997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.773165 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.773199 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.774049 (+ 850us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.774219 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5895,"threads_started":1}
I20260924 01:05:49.774434 24205 raft_consensus.cc:504] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.775729 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a471689c7fe4a798721a2e2bd0836d0" candidate_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a578d3cbc147e29b5e6f604723805c" is_pre_election: true
I20260924 01:05:49.775779 23874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a471689c7fe4a798721a2e2bd0836d0" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
I20260924 01:05:49.775981 24006 raft_consensus.cc:2454] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db6d3a1f06cf435daa8e6a341c4d1dad in term 0.
I20260924 01:05:49.776005 23874 raft_consensus.cc:3037] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:49.776659 23809 leader_election.cc:302] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9a578d3cbc147e29b5e6f604723805c, db6d3a1f06cf435daa8e6a341c4d1dad; no voters:
I20260924 01:05:49.777776 24205 leader_election.cc:288] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 election: Requested vote from peers db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361), a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101)
I20260924 01:05:49.778532 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a471689c7fe4a798721a2e2bd0836d0" candidate_uuid: "859b3ad486c346df91ace7667632a3dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a578d3cbc147e29b5e6f604723805c"
I20260924 01:05:49.778745 24006 raft_consensus.cc:3037] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:05:49.779636 23874 raft_consensus.cc:2454] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 1.
I20260924 01:05:49.780109 24073 leader_election.cc:302] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859b3ad486c346df91ace7667632a3dd, db6d3a1f06cf435daa8e6a341c4d1dad; no voters:
I20260924 01:05:49.780529 24205 raft_consensus.cc:2781] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:05:49.781840 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.761064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:51445 (local address 127.8.254.67:33209)
0924 01:05:49.761272 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.761281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.775011 (+ 13730us) server_negotiation.cc:407] Connection header received
0924 01:05:49.775253 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.775259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.775332 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.775402 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:05:49.775978 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.777088 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.779405 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.780707 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.780838 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.780911 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.781431 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.781567 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:05:49.782495 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.761022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:33209 (local address 127.8.254.65:51445)
0924 01:05:49.774925 (+ 13903us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:49.774944 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:05:49.775116 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:05:49.775612 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:05:49.775617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:05:49.775643 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:05:49.775831 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.775841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.777260 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.777265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.779234 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:05:49.779248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.781060 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.781066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:05:49.781257 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.781289 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:05:49.782041 (+ 752us) client_negotiation.cc:240] Negotiation successful
0924 01:05:49.782223 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13741,"thread_start_us":5394,"threads_started":1}
I20260924 01:05:49.782434 24006 raft_consensus.cc:2454] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859b3ad486c346df91ace7667632a3dd in term 1.
I20260924 01:05:49.783355 24137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a471689c7fe4a798721a2e2bd0836d0" candidate_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859b3ad486c346df91ace7667632a3dd" is_pre_election: true
I20260924 01:05:49.783556 24204 raft_consensus.cc:2741] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 01:05:49.784961 24205 raft_consensus.cc:686] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 1 LEADER]: Becoming Leader. State: Replica: 859b3ad486c346df91ace7667632a3dd, State: Running, Role: LEADER
I20260924 01:05:49.785512 24205 consensus_queue.cc:237] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } }
I20260924 01:05:49.790475 23733 catalog_manager.cc:5495] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 859b3ad486c346df91ace7667632a3dd (127.8.254.67). New cstate: current_term: 1 leader_uuid: "859b3ad486c346df91ace7667632a3dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859b3ad486c346df91ace7667632a3dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 } health_report { overall_health: UNKNOWN } } }
I20260924 01:05:49.869498 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.864027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.8.254.67:33209)
0924 01:05:49.864219 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.864224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.864642 (+ 418us) server_negotiation.cc:407] Connection header received
0924 01:05:49.864874 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.864878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.864980 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.865113 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:05:49.865464 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.866156 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.867528 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.867908 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.868000 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.868446 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:05:49.868973 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:05:49.869078 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.869175 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.869256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:05:49.877519 23874 raft_consensus.cc:1261] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Refusing update from remote peer 859b3ad486c346df91ace7667632a3dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:49.877523 24006 raft_consensus.cc:1261] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Refusing update from remote peer 859b3ad486c346df91ace7667632a3dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:05:49.878521 24205 consensus_queue.cc:1035] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a578d3cbc147e29b5e6f604723805c" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:49.878858 24212 consensus_queue.cc:1035] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6d3a1f06cf435daa8e6a341c4d1dad" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:05:49.890666 24220 mvcc.cc:205] Tried to move back new op lower bound from 7332708146687561728 to 7332708146326343680. Current Snapshot: MvccSnapshot[applied={T|T < 7332708146687561728}]
I20260924 01:05:49.904528 23734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:05:49.904800 23734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:05:49.908035 23734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Sending DeleteTablet for 3 replicas of tablet 1a471689c7fe4a798721a2e2bd0836d0
I20260924 01:05:49.909255 23854 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a471689c7fe4a798721a2e2bd0836d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:49 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:51712
I20260924 01:05:49.909260 24117 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a471689c7fe4a798721a2e2bd0836d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:49 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:45480
I20260924 01:05:49.909308 23986 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a471689c7fe4a798721a2e2bd0836d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:05:49 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:43128
I20260924 01:05:49.910781 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23771
I20260924 01:05:49.921707 24228 tablet_replica.cc:330] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: stopping tablet replica
I20260924 01:05:49.922453 24228 raft_consensus.cc:2229] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:05:49.923157 24228 raft_consensus.cc:2258] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:49.924175 24229 tablet_replica.cc:330] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: stopping tablet replica
I20260924 01:05:49.926192 24228 ts_tablet_manager.cc:1905] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:49.926898 24229 raft_consensus.cc:2229] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:05:49.928793 24230 tablet_replica.cc:330] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: stopping tablet replica
I20260924 01:05:49.929260 24230 raft_consensus.cc:2229] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:05:49.929831 24230 raft_consensus.cc:2258] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:49.931980 24230 ts_tablet_manager.cc:1905] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:49.935324 24229 raft_consensus.cc:2258] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:05:49.936388 24228 ts_tablet_manager.cc:1918] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:49.936677 24228 log.cc:1196] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/1a471689c7fe4a798721a2e2bd0836d0
I20260924 01:05:49.937345 24228 ts_tablet_manager.cc:1939] T 1a471689c7fe4a798721a2e2bd0836d0 P db6d3a1f06cf435daa8e6a341c4d1dad: Deleting consensus metadata
I20260924 01:05:49.937476 24229 ts_tablet_manager.cc:1905] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:05:49.939157 24230 ts_tablet_manager.cc:1918] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:49.939348 24230 log.cc:1196] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/1a471689c7fe4a798721a2e2bd0836d0
I20260924 01:05:49.939411 23721 catalog_manager.cc:4841] TS db6d3a1f06cf435daa8e6a341c4d1dad (127.8.254.65:35361): tablet 1a471689c7fe4a798721a2e2bd0836d0 (table test-table [id=a36eaddab02b48ca9fe4eff6dddd5b64]) successfully deleted
I20260924 01:05:49.939949 24230 ts_tablet_manager.cc:1939] T 1a471689c7fe4a798721a2e2bd0836d0 P a9a578d3cbc147e29b5e6f604723805c: Deleting consensus metadata
I20260924 01:05:49.941684 23720 catalog_manager.cc:4841] TS a9a578d3cbc147e29b5e6f604723805c (127.8.254.66:45101): tablet 1a471689c7fe4a798721a2e2bd0836d0 (table test-table [id=a36eaddab02b48ca9fe4eff6dddd5b64]) successfully deleted
I20260924 01:05:49.945578 24229 ts_tablet_manager.cc:1918] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:05:49.945787 24229 log.cc:1196] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/1a471689c7fe4a798721a2e2bd0836d0
I20260924 01:05:49.946412 24229 ts_tablet_manager.cc:1939] T 1a471689c7fe4a798721a2e2bd0836d0 P 859b3ad486c346df91ace7667632a3dd: Deleting consensus metadata
I20260924 01:05:49.948024 23721 catalog_manager.cc:4841] TS 859b3ad486c346df91ace7667632a3dd (127.8.254.67:33209): tablet 1a471689c7fe4a798721a2e2bd0836d0 (table test-table [id=a36eaddab02b48ca9fe4eff6dddd5b64]) successfully deleted
I20260924 01:05:49.952401 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:49.911708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.8.254.65:35361)
0924 01:05:49.911884 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:49.911890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:49.927937 (+ 16047us) server_negotiation.cc:407] Connection header received
0924 01:05:49.928159 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:49.928168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:49.928266 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:49.928400 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:49.930755 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.932063 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.934296 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:49.934979 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:49.935084 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:49.945007 (+ 9923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:49.945026 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:49.945046 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:49.945127 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:49.949218 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:49.950049 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:49.950062 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:49.950066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:49.950263 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:49.950716 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:49.950721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:49.950725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:49.951235 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:49.951553 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:49.951768 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 01:05:49.952014 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:05:49.953051 23914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:50.045387 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23771
I20260924 01:05:50.071863 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23923
I20260924 01:05:50.088301 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:50.072647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.8.254.66:45101)
0924 01:05:50.072831 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:50.072835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:50.072854 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:05:50.073061 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:50.073065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:50.073139 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:50.073225 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:50.075139 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:50.075868 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:50.077818 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:50.078320 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:50.078429 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:50.081891 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:50.081908 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:50.081923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:50.081969 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:50.084977 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:50.085897 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:50.085908 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:50.085912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:50.086050 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:50.086769 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:50.086776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:50.086780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:50.087224 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:50.087500 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:50.087824 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 01:05:50.087981 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 01:05:50.089038 24046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:50.180761 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23923
I20260924 01:05:50.199657 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 24054
I20260924 01:05:50.218412 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:50.200639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.8.254.67:33209)
0924 01:05:50.200846 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 01:05:50.200851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:05:50.200868 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:05:50.200954 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:05:50.200957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:05:50.201041 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:05:50.201123 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:05:50.203168 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:50.204479 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:50.206399 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:05:50.207200 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:05:50.207346 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:05:50.210649 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:05:50.210697 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:05:50.210720 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:05:50.210770 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:05:50.215082 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:50.215972 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:50.215982 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:50.215986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:50.216162 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:05:50.216584 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:05:50.216590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:05:50.216593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:05:50.217080 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:05:50.217394 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 01:05:50.217789 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 01:05:50.218017 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 01:05:50.219177 24177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:50.343214 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 24054
I20260924 01:05:50.360679 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 23701
I20260924 01:05:50.361840 23764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:05:50.474869 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 23701
I20260924 01:05:50.497237 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40049
--webserver_interface=127.8.254.126
--webserver_port=37663
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:05:50.833717 24236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:50.834031 24236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:50.834234 24236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:50.841640 24236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:05:50.841763 24236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:50.841862 24236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:50.841971 24236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:05:50.842037 24236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:05:50.852349 24236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:40049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=37663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:50.853747 24236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:50.855175 24236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:50.867093 24242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:50.867093 24244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:50.867121 24241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:50.867650 24236 server_base.cc:1034] running on GCE node
I20260924 01:05:50.868685 24236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:50.870318 24236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:50.871570 24236 hybrid_clock.cc:648] HybridClock initialized: now 1790211950871526 us; error 55 us; skew 500 ppm
I20260924 01:05:50.871946 24236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:50 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211950, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:50.875276 24236 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:05:50.876958 24236 webserver.cc:460] Webserver started at http://127.8.254.126:37663/ using document root <none> and password file <none>
I20260924 01:05:50.877704 24236 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:50.877862 24236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:50.883699 24236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 01:05:50.888607 24251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:50.890359 24236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:05:50.890568 24236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "76f1762dc09442f4a565fe4410eebc46"
format_stamp: "Formatted at 2026-09-24 01:05:44 on dist-test-slave-5c7l"
server_key: "7d9921ebedd97e941df0fe6bbf29fc9f"
server_key_iv: "4b57c40e95303d115aacf7a724f033c9"
server_key_version: "encryptionkey@0"
I20260924 01:05:50.891688 24236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:50.925510 24236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:50.929752 24236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:50.930058 24236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:50.950124 24236 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:40049
I20260924 01:05:50.950127 24302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:40049 every 8 connection(s)
I20260924 01:05:50.951830 24236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:05:50.959071 24303 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:05:50.960614 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24236
I20260924 01:05:50.961638 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:35361
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:40049
--webserver_port=43379
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:50.963840 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap starting.
I20260924 01:05:50.986776 24303 log.cc:824] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Log is configured to *not* fsync() on all Append() calls
I20260924 01:05:51.003537 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:05:51.004128 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46: Bootstrap complete.
I20260924 01:05:51.013422 24303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:51.014468 24303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f1762dc09442f4a565fe4410eebc46, State: Initialized, Role: FOLLOWER
I20260924 01:05:51.015194 24303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:51.015440 24303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:05:51.015556 24303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:05:51.015790 24303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 2 FOLLOWER]: Advancing to term 3
I20260924 01:05:51.020252 24303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:51.020817 24303 leader_election.cc:302] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f1762dc09442f4a565fe4410eebc46; no voters:
I20260924 01:05:51.021570 24303 leader_election.cc:288] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 01:05:51.021718 24307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 01:05:51.022921 24307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [term 3 LEADER]: Becoming Leader. State: Replica: 76f1762dc09442f4a565fe4410eebc46, State: Running, Role: LEADER
I20260924 01:05:51.023566 24307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } }
I20260924 01:05:51.023962 24303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:05:51.026520 24308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76f1762dc09442f4a565fe4410eebc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } } }
I20260924 01:05:51.026510 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f1762dc09442f4a565fe4410eebc46. Latest consensus state: current_term: 3 leader_uuid: "76f1762dc09442f4a565fe4410eebc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f1762dc09442f4a565fe4410eebc46" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 40049 } } }
I20260924 01:05:51.026953 24308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:51.026960 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f1762dc09442f4a565fe4410eebc46 [sys.catalog]: This master's current role is: LEADER
I20260924 01:05:51.040984 24312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:05:51.051537 24312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:05:51.053643 24312 catalog_manager.cc:1259] Loaded cluster ID: 00c7990f408246f8af76f10c44345303
I20260924 01:05:51.053856 24312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 01:05:51.056845 24312 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f83e68519ad google::LogMessage::Fail() at ??:0
@ 0x7f83e685488a google::LogMessage::SendToLog() at ??:0
@ 0x7f83e68514de google::LogMessage::Flush() at ??:0
@ 0x7f83e6852a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f83fc284512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f83fc281976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f83fc309610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f83fc309439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f83e949d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f83e94a091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f83e94a0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f83e947beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f83ec7556db start_thread at ??:0
@ 0x7f83e56eb61f clone at ??:0
W20260924 01:05:51.372464 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:51.372784 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:51.372884 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:51.380592 24305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:51.380715 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:51.380833 24305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:05:51.391076 24305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=43379
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:51.392325 24305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:51.393656 24305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:51.407562 24344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:51.407649 24347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:51.407562 24345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:51.408558 24305 server_base.cc:1034] running on GCE node
I20260924 01:05:51.409225 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:51.410270 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:51.411564 24305 hybrid_clock.cc:648] HybridClock initialized: now 1790211951411518 us; error 33 us; skew 500 ppm
I20260924 01:05:51.411950 24305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:51 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211951, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:51.415056 24305 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:05:51.416495 24305 webserver.cc:460] Webserver started at http://127.8.254.65:43379/ using document root <none> and password file <none>
I20260924 01:05:51.417200 24305 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:51.417346 24305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:51.422906 24305 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 01:05:51.425774 24354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:51.427256 24305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 01:05:51.427435 24305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "db6d3a1f06cf435daa8e6a341c4d1dad"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "cd823dd8f79cbeac965db70a6f454b12"
server_key_iv: "b49aa0d682d2112b471b199e3de55dba"
server_key_version: "encryptionkey@0"
I20260924 01:05:51.428490 24305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:51.451256 24305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:51.455367 24305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:51.455657 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:51.456683 24305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:51.458091 24305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:51.458242 24305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:51.458384 24305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:51.458474 24305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:51.494589 24305 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:35361
I20260924 01:05:51.494649 24466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:35361 every 8 connection(s)
I20260924 01:05:51.496623 24305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:05:51.499550 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24305
I20260924 01:05:51.500191 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:45101
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:40049
--webserver_port=34359
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:51.502408 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:51.500241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:48883)
0924 01:05:51.501080 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:51.501369 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":356,"threads_started":1}
W20260924 01:05:51.503067 24467 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:40049 (0 consecutive failures): Network error: Failed to ping master at 127.8.254.126:40049: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
I20260924 01:05:51.504011 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:51.503645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:34181)
0924 01:05:51.503820 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:51.503903 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 01:05:51.505039 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:51.504571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:56107)
0924 01:05:51.504758 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:51.504874 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 01:05:51.505316 24467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:05:51.837007 24470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:51.837320 24470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:51.837504 24470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:51.845078 24470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:51.845206 24470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:51.845423 24470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:05:51.855607 24470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:45101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=34359
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:51.856983 24470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:51.858434 24470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:51.871948 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:51.871960 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:51.871901 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:51.872620 24470 server_base.cc:1034] running on GCE node
I20260924 01:05:51.873266 24470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:51.874311 24470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:51.875520 24470 hybrid_clock.cc:648] HybridClock initialized: now 1790211951875459 us; error 57 us; skew 500 ppm
I20260924 01:05:51.875808 24470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:51 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211951, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:51.879048 24470 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:05:51.880576 24470 webserver.cc:460] Webserver started at http://127.8.254.66:34359/ using document root <none> and password file <none>
I20260924 01:05:51.881408 24470 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:51.881518 24470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:51.887192 24470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 01:05:51.890141 24485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:51.891577 24470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:05:51.891762 24470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "a9a578d3cbc147e29b5e6f604723805c"
format_stamp: "Formatted at 2026-09-24 01:05:45 on dist-test-slave-5c7l"
server_key: "d109b661c5f7b288c92a85f3a68e1682"
server_key_iv: "d67c9c5bd75e0baa27b9c3d6359223a4"
server_key_version: "encryptionkey@0"
I20260924 01:05:51.892786 24470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:51.920516 24470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:51.925241 24470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:51.925479 24470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:51.926491 24470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:51.927834 24470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:51.927917 24470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:51.927990 24470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:51.928047 24470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:51.961941 24470 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:45101
I20260924 01:05:51.962024 24597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:45101 every 8 connection(s)
I20260924 01:05:51.963455 24470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:05:51.964730 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24470
I20260924 01:05:51.965499 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:33209
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:40049
--webserver_port=45561
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:05:51.968923 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:51.967032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:55165)
0924 01:05:51.967837 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:51.968072 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":324,"threads_started":1}
W20260924 01:05:51.969662 24598 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:40049 (0 consecutive failures): Network error: Failed to ping master at 127.8.254.126:40049: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
I20260924 01:05:51.970606 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:51.970287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:39705)
0924 01:05:51.970434 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:51.970504 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 01:05:51.971521 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:51.971216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:45695)
0924 01:05:51.971349 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:51.971418 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 01:05:51.971812 24598 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:05:52.299095 24601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:05:52.299366 24601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:05:52.299554 24601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:05:52.306715 24601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:05:52.306845 24601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:05:52.307018 24601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:05:52.316632 24601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:33209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=45561
--tserver_master_addrs=127.8.254.126:40049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:05:52.317953 24601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:05:52.319267 24601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:05:52.332904 24607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:52.332971 24609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:05:52.333007 24606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:05:52.333747 24601 server_base.cc:1034] running on GCE node
I20260924 01:05:52.334463 24601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:05:52.335399 24601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:05:52.336609 24601 hybrid_clock.cc:648] HybridClock initialized: now 1790211952336561 us; error 30 us; skew 500 ppm
I20260924 01:05:52.336982 24601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:05:52 dist-test-slave-5c7l krb5kdc[23148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211952, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:05:52.340423 24601 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:05:52.341990 24601 webserver.cc:460] Webserver started at http://127.8.254.67:45561/ using document root <none> and password file <none>
I20260924 01:05:52.342784 24601 fs_manager.cc:359] Metadata directory not provided
I20260924 01:05:52.342887 24601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:05:52.348348 24601 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 01:05:52.351464 24616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:52.352901 24601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:05:52.353087 24601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "859b3ad486c346df91ace7667632a3dd"
format_stamp: "Formatted at 2026-09-24 01:05:46 on dist-test-slave-5c7l"
server_key: "1f8a054715319dfc0061da39b4698da0"
server_key_iv: "b3644a5fb92fb1bd2f16a3fdc4949448"
server_key_version: "encryptionkey@0"
I20260924 01:05:52.354146 24601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:05:52.367285 24601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:05:52.371865 24601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:05:52.372166 24601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:05:52.373198 24601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:05:52.374585 24601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:05:52.374725 24601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:52.374858 24601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:05:52.374945 24601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:05:52.408380 24601 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:33209
I20260924 01:05:52.408468 24728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:33209 every 8 connection(s)
I20260924 01:05:52.409897 24601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:05:52.410454 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24601
I20260924 01:05:52.415691 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:52.413844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:47183)
0924 01:05:52.414622 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:52.414834 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":327,"threads_started":1}
W20260924 01:05:52.416311 24729 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:40049 (0 consecutive failures): Network error: Failed to ping master at 127.8.254.126:40049: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
I20260924 01:05:52.417073 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:52.416774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:41919)
0924 01:05:52.416907 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:52.416971 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 01:05:52.417874 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:52.417558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:50043)
0924 01:05:52.417693 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:52.417774 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 01:05:52.418112 24729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 01:05:52.506748 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:52.505937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:37525)
0924 01:05:52.506429 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:52.506520 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":212,"threads_started":1}
I20260924 01:05:52.973219 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:52.972439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:38717)
0924 01:05:52.972905 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:52.972991 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":237,"threads_started":1}
I20260924 01:05:53.419523 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:53.418743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:35375)
0924 01:05:53.419206 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:53.419296 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
I20260924 01:05:53.508373 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:53.507619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:43373)
0924 01:05:53.508120 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:53.508207 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260924 01:05:53.975294 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:53.974439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:39119)
0924 01:05:53.975036 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:53.975124 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":310,"threads_started":1}
I20260924 01:05:54.421109 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:54.420391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:47435)
0924 01:05:54.420867 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:54.420945 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":242,"threads_started":1}
I20260924 01:05:54.510147 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:54.509291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:38409)
0924 01:05:54.509784 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:54.509917 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":257,"threads_started":1}
I20260924 01:05:54.977020 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:54.976241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:59299)
0924 01:05:54.976739 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:54.976836 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":256,"threads_started":1}
I20260924 01:05:55.422993 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:55.422170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:55961)
0924 01:05:55.422729 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:55.422817 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":297,"threads_started":1}
I20260924 01:05:55.511864 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:55.511126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:33111)
0924 01:05:55.511594 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:55.511682 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260924 01:05:55.978896 24743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:55.978068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:34575)
0924 01:05:55.978625 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:55.978723 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":319,"threads_started":1}
I20260924 01:05:56.424566 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:56.423848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:48367)
0924 01:05:56.424293 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:56.424376 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260924 01:05:56.513479 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:56.512765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:57533)
0924 01:05:56.513224 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:56.513308 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
I20260924 01:05:56.981051 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:56.980198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:54537)
0924 01:05:56.980695 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:56.980826 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260924 01:05:57.426172 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:57.425481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:56597)
0924 01:05:57.425909 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:57.425991 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260924 01:05:57.515170 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:57.514466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:49457)
0924 01:05:57.514908 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:57.515000 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":219,"threads_started":1}
I20260924 01:05:57.982887 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:57.982154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:41591)
0924 01:05:57.982634 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:57.982725 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
I20260924 01:05:58.428035 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:58.427194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:48731)
0924 01:05:58.427760 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:58.427864 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":337,"threads_started":1}
I20260924 01:05:58.516973 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:58.516161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:55825)
0924 01:05:58.516654 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:58.516765 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":217,"threads_started":1}
I20260924 01:05:58.984808 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:58.983885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:44395)
0924 01:05:58.984473 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:58.984561 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260924 01:05:59.429812 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:59.428960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:44789)
0924 01:05:59.429436 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:59.429519 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":211,"threads_started":1}
I20260924 01:05:59.518718 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:59.517970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:42987)
0924 01:05:59.518464 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:59.518551 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":281,"threads_started":1}
I20260924 01:05:59.986704 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:05:59.985779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:46335)
0924 01:05:59.986406 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 01:05:59.986520 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":289,"threads_started":1}
I20260924 01:06:00.431727 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:00.430924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:57353)
0924 01:06:00.431401 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:00.431490 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":236,"threads_started":1}
I20260924 01:06:00.520422 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:00.519635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:54031)
0924 01:06:00.520145 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:00.520228 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":261,"threads_started":1}
I20260924 01:06:00.988637 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:00.987762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:60817)
0924 01:06:00.988363 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:00.988454 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":317,"threads_started":1}
I20260924 01:06:01.433766 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:01.432908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:52897)
0924 01:06:01.433484 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:01.433569 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
I20260924 01:06:01.522292 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:01.521452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:59469)
0924 01:06:01.521944 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:01.522032 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":273,"threads_started":1}
I20260924 01:06:01.990622 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:01.989796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:38207)
0924 01:06:01.990375 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:01.990458 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
I20260924 01:06:02.435482 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:02.434764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:35843)
0924 01:06:02.435232 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:02.435319 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260924 01:06:02.524041 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:02.523243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:33007)
0924 01:06:02.523747 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:02.523851 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":248,"threads_started":1}
I20260924 01:06:02.992573 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:02.991749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:55915)
0924 01:06:02.992290 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:02.992391 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":289,"threads_started":1}
I20260924 01:06:03.437165 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:03.436443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:39989)
0924 01:06:03.436895 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:03.436984 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260924 01:06:03.525636 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:03.524976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:55703)
0924 01:06:03.525399 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:03.525479 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":215,"threads_started":1}
I20260924 01:06:03.994486 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:03.993634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:49317)
0924 01:06:03.994155 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:03.994295 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":285,"threads_started":1}
I20260924 01:06:04.438962 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:04.438268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:37565)
0924 01:06:04.438720 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:04.438806 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260924 01:06:04.527484 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:04.526654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:57605)
0924 01:06:04.527156 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:04.527291 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":238,"threads_started":1}
I20260924 01:06:04.996157 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:04.995381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:36177)
0924 01:06:04.995852 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:04.995947 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":216,"threads_started":1}
I20260924 01:06:05.440748 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:05.439932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:42013)
0924 01:06:05.440416 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:05.440532 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260924 01:06:05.529182 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:05.528472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:51961)
0924 01:06:05.528907 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:05.528996 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":221,"threads_started":1}
I20260924 01:06:05.997839 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:05.997107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:50183)
0924 01:06:05.997560 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:05.997658 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260924 01:06:06.442518 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:06.441743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:34429)
0924 01:06:06.442175 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:06.442316 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":208,"threads_started":1}
I20260924 01:06:06.531352 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:06.530451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.65:55663)
0924 01:06:06.531023 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:06.531154 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":327,"threads_started":1}
I20260924 01:06:06.999786 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:06.999044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.66:46025)
0924 01:06:06.999494 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:06.999607 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":230,"threads_started":1}
I20260924 01:06:07.419274 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 24305
I20260924 01:06:07.438181 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:07.420501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.8.254.65:35361)
0924 01:06:07.421169 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:07.421183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:07.421224 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:06:07.422935 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:07.422945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:07.423413 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:07.423716 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:07.425104 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:07.426430 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:07.427676 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:07.428345 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:07.428484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:07.431525 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:07.431535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:07.431552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:07.431591 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:07.434782 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:07.435493 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:07.435500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:07.435507 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:07.435703 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:07.436108 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:07.436111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:07.436113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:07.436659 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:07.436886 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:07.437431 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 01:06:07.437884 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":342,"threads_started":1}
I20260924 01:06:07.439821 24462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:07.444618 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:07.443778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:40049 (local address 127.8.254.67:41245)
0924 01:06:07.444251 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:07.444399 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260924 01:06:07.531695 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 24305
I20260924 01:06:07.549674 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 24470
I20260924 01:06:07.567775 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:07.550815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.8.254.66:45101)
0924 01:06:07.551375 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:07.551388 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:07.551426 (+ 38us) server_negotiation.cc:407] Connection header received
0924 01:06:07.552996 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:07.553006 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:07.553444 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:07.553758 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:07.555136 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:07.556390 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:07.557715 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:07.558558 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:07.558697 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:07.561502 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:07.561510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:07.561527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:07.561569 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:07.564616 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:07.565308 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:07.565315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:07.565323 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:07.565500 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:07.565831 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:07.565834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:07.565836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:07.566460 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:07.566701 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:07.567079 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 01:06:07.567472 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":263,"threads_started":1}
I20260924 01:06:07.569307 24593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:07.653795 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 24470
I20260924 01:06:07.671618 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 24601
I20260924 01:06:07.691262 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:07.672756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.8.254.67:33209)
0924 01:06:07.673450 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:07.673465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:07.673518 (+ 53us) server_negotiation.cc:407] Connection header received
0924 01:06:07.675299 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:07.675309 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:07.675748 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:07.676136 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:07.677744 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:07.679044 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:07.680343 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:07.681016 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:07.681261 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:07.684385 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:07.684393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:07.684409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:07.684454 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:07.687668 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:07.688420 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:07.688428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:07.688435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:07.688602 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:07.688999 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:07.689005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:07.689008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:07.689560 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:07.689886 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:07.690398 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 01:06:07.690945 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":357,"threads_started":1}
I20260924 01:06:07.692719 24724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:07.777998 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 24601
2026-09-24T01:06:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25030 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:06:07 dist-test-slave-5c7l krb5kdc[24786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:06:07 dist-test-slave-5c7l krb5kdc[24786](info): set up 2 sockets
Sep 24 01:06:07 dist-test-slave-5c7l krb5kdc[24786](info): commencing operation
krb5kdc: starting...
W20260924 01:06:08.887981 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:06:09 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:06:09.136013 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39029
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:06:09.478109 24799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:09.478420 24799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:09.478510 24799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:09.485682 24799 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 01:06:09.485781 24799 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 01:06:09.485819 24799 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 01:06:09.485868 24799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:06:09.485916 24799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:09.485985 24799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:09.486024 24799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:06:09.486079 24799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:06:09.498616 24799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:09.499958 24799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:09.501230 24799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:09.513037 24804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:09.512953 24805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:09.513008 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:09.514111 24799 server_base.cc:1034] running on GCE node
I20260924 01:06:09.514935 24799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:09.516350 24799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:09.517581 24799 hybrid_clock.cc:648] HybridClock initialized: now 1790211969517521 us; error 34 us; skew 500 ppm
I20260924 01:06:09.518041 24799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:09 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211969, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:09.528383 24799 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:06:09.530103 24799 webserver.cc:460] Webserver started at http://127.8.254.126:35781/ using document root <none> and password file <none>
I20260924 01:06:09.530887 24799 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:09.531029 24799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:09.531365 24799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:06:09.534570 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "db7a530b79cd42eebacd195d0d0931c5"
format_stamp: "Formatted at 2026-09-24 01:06:09 on dist-test-slave-5c7l"
server_key: "98afc5af56b4ac4ec15b6e996821fc00"
server_key_iv: "b0bc22cdc6e824ba83159cb2e394b45d"
server_key_version: "encryptionkey@0"
I20260924 01:06:09.535521 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "db7a530b79cd42eebacd195d0d0931c5"
format_stamp: "Formatted at 2026-09-24 01:06:09 on dist-test-slave-5c7l"
server_key: "98afc5af56b4ac4ec15b6e996821fc00"
server_key_iv: "b0bc22cdc6e824ba83159cb2e394b45d"
server_key_version: "encryptionkey@0"
I20260924 01:06:09.540809 24799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:06:09.544425 24814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:09.545737 24799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:06:09.545933 24799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "db7a530b79cd42eebacd195d0d0931c5"
format_stamp: "Formatted at 2026-09-24 01:06:09 on dist-test-slave-5c7l"
server_key: "98afc5af56b4ac4ec15b6e996821fc00"
server_key_iv: "b0bc22cdc6e824ba83159cb2e394b45d"
server_key_version: "encryptionkey@0"
I20260924 01:06:09.546164 24799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:09.596534 24799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:09.600693 24799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:09.600988 24799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:09.619886 24799 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39029
I20260924 01:06:09.619905 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39029 every 8 connection(s)
I20260924 01:06:09.621371 24799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:06:09.625229 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:09.631058 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24799
I20260924 01:06:09.631302 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:06:09.632164 9209 external_mini_cluster.cc:1428] Setting key b285ef857c9e8664eb7144b3420bd62a
I20260924 01:06:09.637425 24866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap starting.
I20260924 01:06:09.641280 24866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:09.642552 24866 log.cc:824] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:09.645852 24866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: No bootstrap required, opened a new log
Sep 24 01:06:09 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:09.655378 24866 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:09.655829 24866 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:09.655943 24866 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7a530b79cd42eebacd195d0d0931c5, State: Initialized, Role: FOLLOWER
I20260924 01:06:09.656672 24866 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:09.656947 24866 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:06:09.657090 24866 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:06:09.657289 24866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:09.658129 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:09.634692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.8.254.126:39029)
0924 01:06:09.636000 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:09.636012 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:09.636054 (+ 42us) server_negotiation.cc:407] Connection header received
0924 01:06:09.637787 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:09.637811 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:09.638298 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:09.638637 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:09.640483 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:09.641856 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:09.643664 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:09.644401 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:09.644534 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:09.649734 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:09.649742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:09.649760 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:09.649804 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:09.653412 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:09.654396 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:09.654404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:09.654411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:09.654599 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:09.655029 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:09.655033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:09.655034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:09.655573 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:09.655886 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:09.656397 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 01:06:09.656859 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":795,"threads_started":1}
I20260924 01:06:09.659782 24866 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:09.660295 24866 leader_election.cc:302] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7a530b79cd42eebacd195d0d0931c5; no voters:
I20260924 01:06:09.661092 24866 leader_election.cc:288] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:06:09.661206 24871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:06:09.661993 24871 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 LEADER]: Becoming Leader. State: Replica: db7a530b79cd42eebacd195d0d0931c5, State: Running, Role: LEADER
I20260924 01:06:09.662612 24871 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:09.663470 24866 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:06:09.665602 24872 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7a530b79cd42eebacd195d0d0931c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } } }
I20260924 01:06:09.665659 24873 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7a530b79cd42eebacd195d0d0931c5. Latest consensus state: current_term: 1 leader_uuid: "db7a530b79cd42eebacd195d0d0931c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } } }
I20260924 01:06:09.666136 24873 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: This master's current role is: LEADER
I20260924 01:06:09.667452 24872 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: This master's current role is: LEADER
I20260924 01:06:09.668358 24876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:06:09.673525 24876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:06:09.682622 24876 catalog_manager.cc:1347] Generated new cluster ID: 5557d5c986d9479daad155c6507b1f2e
I20260924 01:06:09.682747 24876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:06:09.693871 24876 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:06:09.695299 24876 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:06:09.701646 24876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Generated new TSK 0
I20260924 01:06:09.702540 24876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:06:09.766919 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39029
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:06:10.110813 24892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:10.111150 24892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:10.111323 24892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:10.118657 24892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:10.118803 24892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:10.118981 24892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:06:10.128999 24892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:10.130450 24892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:10.131842 24892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:10.144783 24897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:10.144870 24898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:10.144814 24900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:10.145855 24892 server_base.cc:1034] running on GCE node
I20260924 01:06:10.146559 24892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:10.147552 24892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:10.148751 24892 hybrid_clock.cc:648] HybridClock initialized: now 1790211970148709 us; error 34 us; skew 500 ppm
I20260924 01:06:10.149089 24892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:10 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211970, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:10.152505 24892 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:06:10.154050 24892 webserver.cc:460] Webserver started at http://127.8.254.65:40765/ using document root <none> and password file <none>
I20260924 01:06:10.154938 24892 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:10.155144 24892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:10.155640 24892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:06:10.158819 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "315abe3fc0c740779cf2dc36359464ef"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "b2e1a9859053377955aaac9275367e1f"
server_key_iv: "e0b069b1fcbfd6539878d7e5aaa09393"
server_key_version: "encryptionkey@0"
I20260924 01:06:10.159631 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "315abe3fc0c740779cf2dc36359464ef"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "b2e1a9859053377955aaac9275367e1f"
server_key_iv: "e0b069b1fcbfd6539878d7e5aaa09393"
server_key_version: "encryptionkey@0"
I20260924 01:06:10.165151 24892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:06:10.168920 24907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:10.170455 24892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 01:06:10.170635 24892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "315abe3fc0c740779cf2dc36359464ef"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "b2e1a9859053377955aaac9275367e1f"
server_key_iv: "e0b069b1fcbfd6539878d7e5aaa09393"
server_key_version: "encryptionkey@0"
I20260924 01:06:10.170840 24892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:10.198010 24892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:10.202283 24892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:10.202577 24892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:10.203580 24892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:10.204910 24892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:10.204982 24892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:10.205155 24892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:10.205216 24892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:10.238636 24892 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:34787
I20260924 01:06:10.238673 25019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:34787 every 8 connection(s)
I20260924 01:06:10.239974 24892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:06:10.244410 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 24892
I20260924 01:06:10.244622 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:06:10.245260 9209 external_mini_cluster.cc:1428] Setting key 98cb83afba791d537f8086b85f1c5435
Sep 24 01:06:10 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211970, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:10.260541 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:10.242685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:55113 (local address 127.8.254.126:39029)
0924 01:06:10.243186 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:10.243190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:10.243997 (+ 807us) server_negotiation.cc:407] Connection header received
0924 01:06:10.246015 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:10.246021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:10.246116 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:10.246271 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:10.247789 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.248516 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.250079 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.250588 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.250735 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:10.254626 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:10.254632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:10.254635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:10.254670 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:10.257336 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:10.258096 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:10.258108 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:10.258110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:10.258298 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:10.258745 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:10.258751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:10.258754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:10.259026 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:10.259183 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:10.259914 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 01:06:10.260149 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
I20260924 01:06:10.261665 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:10.243100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:55113)
0924 01:06:10.243817 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:10.243867 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:10.245727 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:10.246515 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:10.246523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:10.247027 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:06:10.247611 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:10.247629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.248686 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.248690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:10.249917 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:10.249934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.250836 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.250846 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:10.251088 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:10.251817 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:06:10.251839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:06:10.254388 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:06:10.257491 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:10.257501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:10.257519 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:10.257961 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:10.258412 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:10.258416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:10.258420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:10.258619 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:10.259169 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:06:10.259176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:06:10.259532 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 01:06:10.259815 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 01:06:10.260394 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260924 01:06:10.263447 25020 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39029
I20260924 01:06:10.263797 25020 heartbeater.cc:464] Registering TS with master...
I20260924 01:06:10.264779 25020 heartbeater.cc:511] Master 127.8.254.126:39029 requested a full tablet report, sending...
I20260924 01:06:10.267480 24831 ts_manager.cc:194] Registered new tserver with Master: 315abe3fc0c740779cf2dc36359464ef (127.8.254.65:34787)
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:06:10.269712 24831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:55113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:06:10.301592 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39029
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:06:10.641191 25026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:10.641580 25026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:10.641736 25026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:10.650144 25026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:10.650314 25026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:10.650483 25026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:06:10.660451 25026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:10.661791 25026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:10.663053 25026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:10.676304 25032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:10.676471 25031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:10.676841 25034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:10.677410 25026 server_base.cc:1034] running on GCE node
I20260924 01:06:10.678056 25026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:10.679080 25026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:10.680290 25026 hybrid_clock.cc:648] HybridClock initialized: now 1790211970680243 us; error 36 us; skew 500 ppm
I20260924 01:06:10.680681 25026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:10 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211970, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:10.683933 25026 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:06:10.685590 25026 webserver.cc:460] Webserver started at http://127.8.254.66:42861/ using document root <none> and password file <none>
I20260924 01:06:10.686372 25026 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:10.686515 25026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:10.686854 25026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:06:10.689771 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "83e1014bc297e7418ba5476fe80ea272"
server_key_iv: "99b7eaa7eec140dcb94ef40707b9cad1"
server_key_version: "encryptionkey@0"
I20260924 01:06:10.690620 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "83e1014bc297e7418ba5476fe80ea272"
server_key_iv: "99b7eaa7eec140dcb94ef40707b9cad1"
server_key_version: "encryptionkey@0"
I20260924 01:06:10.695947 25026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 01:06:10.699468 25041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:10.700826 25026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:06:10.701025 25026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "83e1014bc297e7418ba5476fe80ea272"
server_key_iv: "99b7eaa7eec140dcb94ef40707b9cad1"
server_key_version: "encryptionkey@0"
I20260924 01:06:10.701232 25026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:10.712563 25026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:10.716741 25026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:10.717061 25026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:10.718058 25026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:10.719493 25026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:10.719601 25026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:10.719749 25026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:10.719823 25026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:10.753494 25026 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:35423
I20260924 01:06:10.753535 25153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:35423 every 8 connection(s)
I20260924 01:06:10.754925 25026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:06:10.756209 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25026
I20260924 01:06:10.756420 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:06:10.757010 9209 external_mini_cluster.cc:1428] Setting key a9cb2b61e8bdcd6ba18f6d45c2248858
Sep 24 01:06:10 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211970, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:10.778985 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:10.758306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:57093 (local address 127.8.254.126:39029)
0924 01:06:10.758474 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:10.758478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:10.759511 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 01:06:10.762542 (+ 3031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:10.762550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:10.762638 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:10.762767 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:10.764924 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.765621 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.768192 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.768706 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.768852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:10.772736 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:10.772741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:10.772744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:10.772779 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:10.775577 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:10.776567 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:10.776570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:10.776573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:10.776706 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:10.777197 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:10.777200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:10.777202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:10.777480 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:10.777597 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:10.778565 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 01:06:10.778741 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:06:10.780668 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:10.758456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:57093)
0924 01:06:10.759332 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:10.759380 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:10.762219 (+ 2839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:10.763038 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:10.763052 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:10.763887 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:06:10.764751 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:10.764779 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.765788 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.765796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:10.768034 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:10.768058 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:10.768933 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:10.768938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:10.769138 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:10.769905 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:06:10.769930 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:06:10.772495 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:06:10.775743 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:10.775755 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:10.775776 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:10.776443 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:10.776818 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:10.776825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:10.776832 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:10.777082 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:10.777619 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:06:10.777629 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:06:10.778098 (+ 469us) client_negotiation.cc:769] Sending connection context
0924 01:06:10.778479 (+ 381us) client_negotiation.cc:240] Negotiation successful
0924 01:06:10.779094 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":347,"threads_started":1}
I20260924 01:06:10.782279 25154 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39029
I20260924 01:06:10.782644 25154 heartbeater.cc:464] Registering TS with master...
I20260924 01:06:10.783591 25154 heartbeater.cc:511] Master 127.8.254.126:39029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
I20260924 01:06:10.785353 24831 ts_manager.cc:194] Registered new tserver with Master: e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66:35423)
I20260924 01:06:10.786173 24831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:57093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:06:10.818173 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39029
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:06:11.150485 25159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:11.150821 25159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:11.150965 25159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:11.158383 25159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:11.158495 25159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:11.158618 25159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:06:11.168506 25159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:11.169786 25159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:11.171132 25159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:11.184900 25164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:11.184939 25165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:11.184983 25167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:11.185797 25159 server_base.cc:1034] running on GCE node
I20260924 01:06:11.186461 25159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:11.187366 25159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:11.188561 25159 hybrid_clock.cc:648] HybridClock initialized: now 1790211971188514 us; error 35 us; skew 500 ppm
I20260924 01:06:11.188896 25159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:11 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:11.192272 25159 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:06:11.193667 25159 webserver.cc:460] Webserver started at http://127.8.254.67:45345/ using document root <none> and password file <none>
I20260924 01:06:11.194397 25159 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:11.194530 25159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:11.194854 25159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:06:11.197634 25159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "2507eb95da124322a0fe41d4cce79ef8"
format_stamp: "Formatted at 2026-09-24 01:06:11 on dist-test-slave-5c7l"
server_key: "bb2cd1c28b66c1677da195981847b817"
server_key_iv: "7feeebf9126bf38b183db62dcfd36a90"
server_key_version: "encryptionkey@0"
I20260924 01:06:11.198453 25159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "2507eb95da124322a0fe41d4cce79ef8"
format_stamp: "Formatted at 2026-09-24 01:06:11 on dist-test-slave-5c7l"
server_key: "bb2cd1c28b66c1677da195981847b817"
server_key_iv: "7feeebf9126bf38b183db62dcfd36a90"
server_key_version: "encryptionkey@0"
I20260924 01:06:11.203450 25159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 01:06:11.206955 25174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:11.208345 25159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:06:11.208518 25159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "2507eb95da124322a0fe41d4cce79ef8"
format_stamp: "Formatted at 2026-09-24 01:06:11 on dist-test-slave-5c7l"
server_key: "bb2cd1c28b66c1677da195981847b817"
server_key_iv: "7feeebf9126bf38b183db62dcfd36a90"
server_key_version: "encryptionkey@0"
I20260924 01:06:11.208710 25159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:11.224169 25159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:11.228440 25159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:11.228761 25159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:11.229715 25159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:11.231112 25159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:11.231217 25159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:11.231367 25159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:11.231441 25159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:11.266507 25159 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:42615
I20260924 01:06:11.266553 25286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:42615 every 8 connection(s)
I20260924 01:06:11.267869 25159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:06:11.272979 25020 heartbeater.cc:503] Master 127.8.254.126:39029 was elected leader, sending a full tablet report...
I20260924 01:06:11.273721 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25159
I20260924 01:06:11.273917 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:06:11.274521 9209 external_mini_cluster.cc:1428] Setting key 9106fbe8a14ceb4d578bbfb2326d923d
Sep 24 01:06:11 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:11.290267 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.270690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:56625 (local address 127.8.254.126:39029)
0924 01:06:11.270880 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.270885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.272169 (+ 1284us) server_negotiation.cc:407] Connection header received
0924 01:06:11.274563 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.274571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.274666 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.274788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:11.276536 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.277472 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.279504 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.280185 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.280314 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.284387 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:11.284393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:11.284396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:11.284429 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:11.286977 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.287689 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.287712 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.287715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.287878 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.288276 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.288280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.288284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.288680 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:11.288822 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.289667 (+ 845us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.289909 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:06:11.291299 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.271128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:56625)
0924 01:06:11.272002 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:11.272050 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:11.274146 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:11.275078 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:11.275090 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:11.275798 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:06:11.276380 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.276397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.277632 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.277636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.279300 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.279315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.280664 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.280674 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.280995 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.281823 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:06:11.281863 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:06:11.284198 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:06:11.287110 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:11.287117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:11.287128 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:11.287547 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:11.287992 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:11.287996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:11.287998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:11.288177 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:11.288818 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:06:11.288824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:06:11.289226 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 01:06:11.289522 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 01:06:11.290040 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":362,"threads_started":1}
I20260924 01:06:11.292734 25287 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39029
I20260924 01:06:11.293010 25287 heartbeater.cc:464] Registering TS with master...
I20260924 01:06:11.293708 25287 heartbeater.cc:511] Master 127.8.254.126:39029 requested a full tablet report, sending...
I20260924 01:06:11.295120 24831 ts_manager.cc:194] Registered new tserver with Master: 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615)
I20260924 01:06:11.295890 24831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:56625
I20260924 01:06:11.302126 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:06:11.320494 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.306957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.8.254.126:39029)
0924 01:06:11.307115 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.307119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.307570 (+ 451us) server_negotiation.cc:407] Connection header received
0924 01:06:11.307772 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.307777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.307856 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.307962 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:11.309306 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.310082 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.311551 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.312079 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.312218 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.314953 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:11.314960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:11.314964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:11.315008 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:11.317566 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.318371 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.318376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.318380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.318525 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.318926 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.318931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.318934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.319278 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:11.319551 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.319888 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.320152 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 01:06:11.324770 24831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:06:11.331799 24831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:06:11.364523 25299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.344781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:56664)
0924 01:06:11.345821 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:11.345893 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:11.346116 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:11.356997 (+ 10881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:11.357002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:11.357030 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:11.357369 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.357379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.359413 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.359418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.361728 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.361740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.363324 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.363332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.363576 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.363638 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 01:06:11.363749 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 01:06:11.363987 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":80,"thread_start_us":508,"threads_started":1}
I20260924 01:06:11.365069 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.345011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.8.254.65:34787)
0924 01:06:11.356274 (+ 11263us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.356283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.356313 (+ 30us) server_negotiation.cc:407] Connection header received
0924 01:06:11.356416 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.356426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.356717 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.357009 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:11.357582 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.359270 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.361919 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.363050 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.364303 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.364564 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.364639 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.364823 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11067,"spinlock_wait_cycles":25856,"thread_start_us":11047,"threads_started":1}
I20260924 01:06:11.365058 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.346490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.8.254.67:42615)
0924 01:06:11.355455 (+ 8965us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.355467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.355512 (+ 45us) server_negotiation.cc:407] Connection header received
0924 01:06:11.355625 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.355636 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.355964 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.356173 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:11.357151 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.358875 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.361870 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.363303 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.363422 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.363740 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.364289 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.364472 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8770,"thread_start_us":8594,"threads_started":1}
I20260924 01:06:11.365351 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.347078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.8.254.66:35423)
0924 01:06:11.356879 (+ 9801us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.356890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.356920 (+ 30us) server_negotiation.cc:407] Connection header received
0924 01:06:11.357018 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.357028 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.357272 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.357478 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:11.358129 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.359616 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.361545 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.363040 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.363194 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.363426 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.364516 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.364700 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9615,"thread_start_us":9457,"threads_started":1}
I20260924 01:06:11.365574 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.346983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:35423 (local address 127.0.0.1:44950)
0924 01:06:11.355819 (+ 8836us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:11.355837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:11.355958 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:11.357482 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:11.357490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:11.357519 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:11.357917 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.357929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.359766 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.359773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.361367 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.361381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.363595 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.363602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.363796 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.363841 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 01:06:11.363933 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 01:06:11.364083 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8700,"mutex_wait_us":1,"thread_start_us":8529,"threads_started":1}
I20260924 01:06:11.366050 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.344874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:42615 (local address 127.0.0.1:46598)
0924 01:06:11.346621 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:11.346636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:11.346749 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:11.356447 (+ 9698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:11.356457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:11.356513 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:11.356933 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.356949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.359048 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.359053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.361683 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.361700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.363879 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.363888 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.364066 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.364094 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 01:06:11.365361 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0924 01:06:11.365538 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"mutex_wait_us":20,"thread_start_us":1494,"threads_started":1}
I20260924 01:06:11.368906 24955 tablet_service.cc:1467] Processing CreateTablet for tablet d54d3022fd8f4a22ac7ad50474240492 (DEFAULT_TABLE table=test-table [id=4f7cee2d5e9f40659cc2a356a0d8978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:06:11.369599 25222 tablet_service.cc:1467] Processing CreateTablet for tablet d54d3022fd8f4a22ac7ad50474240492 (DEFAULT_TABLE table=test-table [id=4f7cee2d5e9f40659cc2a356a0d8978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:06:11.369549 25089 tablet_service.cc:1467] Processing CreateTablet for tablet d54d3022fd8f4a22ac7ad50474240492 (DEFAULT_TABLE table=test-table [id=4f7cee2d5e9f40659cc2a356a0d8978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:06:11.370589 24955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54d3022fd8f4a22ac7ad50474240492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:11.371001 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54d3022fd8f4a22ac7ad50474240492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:11.371001 25222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54d3022fd8f4a22ac7ad50474240492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:11.382897 25305 tablet_bootstrap.cc:492] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Bootstrap starting.
I20260924 01:06:11.385622 25306 tablet_bootstrap.cc:492] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Bootstrap starting.
I20260924 01:06:11.386683 25307 tablet_bootstrap.cc:492] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Bootstrap starting.
I20260924 01:06:11.387980 25305 tablet_bootstrap.cc:654] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:11.389168 25306 tablet_bootstrap.cc:654] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:11.389336 25305 log.cc:824] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:11.389599 25307 tablet_bootstrap.cc:654] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:11.390429 25306 log.cc:824] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:11.390897 25307 log.cc:824] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:11.392657 25305 tablet_bootstrap.cc:492] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: No bootstrap required, opened a new log
I20260924 01:06:11.392923 25305 ts_tablet_manager.cc:1397] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 01:06:11.393239 25306 tablet_bootstrap.cc:492] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: No bootstrap required, opened a new log
I20260924 01:06:11.393517 25306 ts_tablet_manager.cc:1397] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 01:06:11.394027 25307 tablet_bootstrap.cc:492] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: No bootstrap required, opened a new log
I20260924 01:06:11.394311 25307 ts_tablet_manager.cc:1397] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 01:06:11.401181 25305 raft_consensus.cc:348] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.401595 25305 raft_consensus.cc:374] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:11.401762 25305 raft_consensus.cc:729] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315abe3fc0c740779cf2dc36359464ef, State: Initialized, Role: FOLLOWER
I20260924 01:06:11.402436 25305 consensus_queue.cc:260] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.404325 25305 ts_tablet_manager.cc:1428] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 01:06:11.404238 25306 raft_consensus.cc:348] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.404767 25306 raft_consensus.cc:374] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:11.404925 25306 raft_consensus.cc:729] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d3ba02424340fd9e2a70f5b1aa6aa3, State: Initialized, Role: FOLLOWER
I20260924 01:06:11.405009 25307 raft_consensus.cc:348] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.405455 25307 raft_consensus.cc:374] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:11.405561 25307 raft_consensus.cc:729] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2507eb95da124322a0fe41d4cce79ef8, State: Initialized, Role: FOLLOWER
I20260924 01:06:11.405637 25306 consensus_queue.cc:260] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.406170 25307 consensus_queue.cc:260] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.407699 25306 ts_tablet_manager.cc:1428] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260924 01:06:11.407780 25154 heartbeater.cc:503] Master 127.8.254.126:39029 was elected leader, sending a full tablet report...
I20260924 01:06:11.408839 25287 heartbeater.cc:503] Master 127.8.254.126:39029 was elected leader, sending a full tablet report...
I20260924 01:06:11.409432 25307 ts_tablet_manager.cc:1428] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260924 01:06:11.492558 25021 tablet.cc:2369] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:06:11.506903 25155 tablet.cc:2369] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:06:11.519532 25288 tablet.cc:2369] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:06:11.578231 25312 raft_consensus.cc:482] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:06:11.578521 25312 raft_consensus.cc:504] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.580412 25312 leader_election.cc:288] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315abe3fc0c740779cf2dc36359464ef (127.8.254.65:34787), 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615)
I20260924 01:06:11.587842 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.581005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:56449 (local address 127.8.254.67:42615)
0924 01:06:11.581195 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.581203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.581804 (+ 601us) server_negotiation.cc:407] Connection header received
0924 01:06:11.581976 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.581980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.582037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.582148 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:11.582883 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.583590 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.585948 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.586807 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.586931 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.587015 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.587432 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.587563 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 01:06:11.587862 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.580861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:37595 (local address 127.8.254.65:34787)
0924 01:06:11.581187 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.581196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.581678 (+ 482us) server_negotiation.cc:407] Connection header received
0924 01:06:11.581889 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.581896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.582001 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.582070 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:11.582808 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.583557 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.585997 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.586906 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.587072 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.587157 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.587469 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.587592 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 01:06:11.588050 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.580706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:42615 (local address 127.8.254.66:56449)
0924 01:06:11.581671 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:11.581691 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:11.581851 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:11.582218 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:11.582225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:11.582268 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:11.582681 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.582692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.583778 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.583782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.585796 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.585810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.587039 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.587046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.587266 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.587296 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 01:06:11.587373 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 01:06:11.587531 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":17,"thread_start_us":650,"threads_started":1}
I20260924 01:06:11.588104 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.580661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.8.254.66:37595)
0924 01:06:11.581553 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:11.581578 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:11.581707 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:11.582119 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:11.582127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:11.582153 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:11.582648 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.582662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.583689 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.583698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.585807 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:11.585819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.587138 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.587143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:11.587292 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.587314 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 01:06:11.587426 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 01:06:11.587564 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":64,"thread_start_us":534,"threads_started":1}
I20260924 01:06:11.589228 25242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54d3022fd8f4a22ac7ad50474240492" candidate_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2507eb95da124322a0fe41d4cce79ef8" is_pre_election: true
I20260924 01:06:11.589242 24975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54d3022fd8f4a22ac7ad50474240492" candidate_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315abe3fc0c740779cf2dc36359464ef" is_pre_election: true
I20260924 01:06:11.589829 25242 raft_consensus.cc:2454] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d3ba02424340fd9e2a70f5b1aa6aa3 in term 0.
I20260924 01:06:11.589829 24975 raft_consensus.cc:2454] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d3ba02424340fd9e2a70f5b1aa6aa3 in term 0.
I20260924 01:06:11.590603 25042 leader_election.cc:302] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315abe3fc0c740779cf2dc36359464ef, e8d3ba02424340fd9e2a70f5b1aa6aa3; no voters:
I20260924 01:06:11.591085 25312 raft_consensus.cc:2781] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:06:11.591243 25312 raft_consensus.cc:482] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:06:11.591373 25312 raft_consensus.cc:3037] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:11.593647 25312 raft_consensus.cc:504] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.594342 25312 leader_election.cc:288] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [CANDIDATE]: Term 1 election: Requested vote from peers 315abe3fc0c740779cf2dc36359464ef (127.8.254.65:34787), 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615)
I20260924 01:06:11.594844 25242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54d3022fd8f4a22ac7ad50474240492" candidate_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2507eb95da124322a0fe41d4cce79ef8"
I20260924 01:06:11.594846 24975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54d3022fd8f4a22ac7ad50474240492" candidate_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315abe3fc0c740779cf2dc36359464ef"
I20260924 01:06:11.595058 24975 raft_consensus.cc:3037] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:11.595042 25242 raft_consensus.cc:3037] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:11.597285 25242 raft_consensus.cc:2454] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d3ba02424340fd9e2a70f5b1aa6aa3 in term 1.
I20260924 01:06:11.597519 24975 raft_consensus.cc:2454] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d3ba02424340fd9e2a70f5b1aa6aa3 in term 1.
I20260924 01:06:11.597730 25045 leader_election.cc:302] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2507eb95da124322a0fe41d4cce79ef8, e8d3ba02424340fd9e2a70f5b1aa6aa3; no voters:
I20260924 01:06:11.598044 25312 raft_consensus.cc:2781] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:06:11.598541 25312 raft_consensus.cc:686] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 LEADER]: Becoming Leader. State: Replica: e8d3ba02424340fd9e2a70f5b1aa6aa3, State: Running, Role: LEADER
I20260924 01:06:11.599231 25312 consensus_queue.cc:237] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } }
I20260924 01:06:11.603727 24830 catalog_manager.cc:5495] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } health_report { overall_health: HEALTHY } } }
I20260924 01:06:11.655740 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.650063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.8.254.66:35423)
0924 01:06:11.650260 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.650265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.650284 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:06:11.650470 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.650474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.650554 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.650701 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:06:11.651205 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.651918 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.653393 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.653812 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.653911 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.654618 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:06:11.655196 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:06:11.655315 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.655418 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.655486 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 01:06:11.664193 25242 raft_consensus.cc:1261] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Refusing update from remote peer e8d3ba02424340fd9e2a70f5b1aa6aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:06:11.664213 24975 raft_consensus.cc:1261] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Refusing update from remote peer e8d3ba02424340fd9e2a70f5b1aa6aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:06:11.665303 25316 consensus_queue.cc:1035] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:06:11.665637 25312 consensus_queue.cc:1035] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:06:11.685621 25321 mvcc.cc:205] Tried to move back new op lower bound from 7332708235926052864 to 7332708235674284032. Current Snapshot: MvccSnapshot[applied={T|T < 7332708235926052864}]
I20260924 01:06:11.691020 24830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:06:11.691345 24830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:06:11.695001 24830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Sending DeleteTablet for 3 replicas of tablet d54d3022fd8f4a22ac7ad50474240492
I20260924 01:06:11.696269 25222 tablet_service.cc:1514] Processing DeleteTablet for tablet d54d3022fd8f4a22ac7ad50474240492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:06:11 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:46598
I20260924 01:06:11.696272 24955 tablet_service.cc:1514] Processing DeleteTablet for tablet d54d3022fd8f4a22ac7ad50474240492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:06:11 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:56664
I20260924 01:06:11.697033 25089 tablet_service.cc:1514] Processing DeleteTablet for tablet d54d3022fd8f4a22ac7ad50474240492 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:06:11 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:44950
I20260924 01:06:11.697168 25329 tablet_replica.cc:330] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: stopping tablet replica
I20260924 01:06:11.697188 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 24892
I20260924 01:06:11.697603 25329 raft_consensus.cc:2229] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:06:11.698101 25329 raft_consensus.cc:2258] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:06:11.698647 25328 tablet_replica.cc:330] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: stopping tablet replica
I20260924 01:06:11.699038 25328 raft_consensus.cc:2229] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:06:11.699522 25328 raft_consensus.cc:2258] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:06:11.701962 25328 ts_tablet_manager.cc:1905] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:11.707096 25329 ts_tablet_manager.cc:1905] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:11.709371 25328 ts_tablet_manager.cc:1918] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:06:11.709589 25328 log.cc:1196] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/d54d3022fd8f4a22ac7ad50474240492
I20260924 01:06:11.710147 25328 ts_tablet_manager.cc:1939] T d54d3022fd8f4a22ac7ad50474240492 P 2507eb95da124322a0fe41d4cce79ef8: Deleting consensus metadata
I20260924 01:06:11.712531 24818 catalog_manager.cc:4841] TS 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615): tablet d54d3022fd8f4a22ac7ad50474240492 (table test-table [id=4f7cee2d5e9f40659cc2a356a0d8978d]) successfully deleted
I20260924 01:06:11.715700 25330 tablet_replica.cc:330] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: stopping tablet replica
I20260924 01:06:11.716161 25330 raft_consensus.cc:2229] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:06:11.716869 25330 raft_consensus.cc:2258] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:06:11.719177 25330 ts_tablet_manager.cc:1905] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:11.719880 25329 ts_tablet_manager.cc:1918] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:06:11.720109 25329 log.cc:1196] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/d54d3022fd8f4a22ac7ad50474240492
I20260924 01:06:11.720762 25329 ts_tablet_manager.cc:1939] T d54d3022fd8f4a22ac7ad50474240492 P 315abe3fc0c740779cf2dc36359464ef: Deleting consensus metadata
I20260924 01:06:11.722813 24815 catalog_manager.cc:4841] TS 315abe3fc0c740779cf2dc36359464ef (127.8.254.65:34787): tablet d54d3022fd8f4a22ac7ad50474240492 (table test-table [id=4f7cee2d5e9f40659cc2a356a0d8978d]) successfully deleted
Sep 24 01:06:11 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:06:11.726684 25330 ts_tablet_manager.cc:1918] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:06:11.726877 25330 log.cc:1196] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/d54d3022fd8f4a22ac7ad50474240492
I20260924 01:06:11.727443 25330 ts_tablet_manager.cc:1939] T d54d3022fd8f4a22ac7ad50474240492 P e8d3ba02424340fd9e2a70f5b1aa6aa3: Deleting consensus metadata
I20260924 01:06:11.729274 24815 catalog_manager.cc:4841] TS e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66:35423): tablet d54d3022fd8f4a22ac7ad50474240492 (table test-table [id=4f7cee2d5e9f40659cc2a356a0d8978d]) successfully deleted
I20260924 01:06:11.733038 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.698519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.8.254.65:34787)
0924 01:06:11.698729 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.698736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.714781 (+ 16045us) server_negotiation.cc:407] Connection header received
0924 01:06:11.715027 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.715036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.715150 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.715287 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:11.717063 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.718345 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.720309 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.721044 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.721145 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.726454 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:11.726468 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:11.726483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:11.726521 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:11.729920 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.730778 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.730788 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.730791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.730942 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.731372 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.731377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.731383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.731753 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:11.732071 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.732447 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.732674 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:06:11.733754 25015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:11.846235 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 24892
I20260924 01:06:11.870419 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25026
Sep 24 01:06:11 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:06:11.888950 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:11.871316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.8.254.66:35423)
0924 01:06:11.871491 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:11.871496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:11.871553 (+ 57us) server_negotiation.cc:407] Connection header received
0924 01:06:11.871755 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:11.871759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:11.871826 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:11.871913 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:11.873666 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.874584 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.876641 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:11.877097 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:11.877196 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:11.882849 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:11.882869 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:11.882885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:11.882932 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:11.886057 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.886893 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.886900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.886918 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.887110 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:11.887510 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:11.887514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:11.887516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:11.887871 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:11.888119 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:11.888469 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 01:06:11.888630 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 01:06:11.889634 25149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:12.005728 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25026
I20260924 01:06:12.025276 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25159
Sep 24 01:06:12 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:06:12.041163 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:12.026032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.8.254.67:42615)
0924 01:06:12.026261 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:12.026268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:12.026373 (+ 105us) server_negotiation.cc:407] Connection header received
0924 01:06:12.026596 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:12.026601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:12.026668 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:12.026776 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:12.028129 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:12.028845 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:12.030383 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:12.030754 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:12.030857 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:12.035359 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:12.035376 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:12.035392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:12.035432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:12.038298 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:12.039029 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:12.039037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:12.039039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:12.039180 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:12.039587 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:12.039591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:12.039593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:12.039941 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:12.040239 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:12.040572 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 01:06:12.040799 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 01:06:12.041818 25282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:12.133857 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25159
I20260924 01:06:12.151242 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 24799
I20260924 01:06:12.152434 24861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:12.265132 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 24799
I20260924 01:06:32.288820 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39029
--webserver_interface=127.8.254.126
--webserver_port=35781
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:06:32.625499 25336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:32.625859 25336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:32.626015 25336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:32.633340 25336 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 01:06:32.633494 25336 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 01:06:32.633589 25336 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 01:06:32.633697 25336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:06:32.633797 25336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:32.633914 25336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:32.634001 25336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:06:32.634081 25336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:06:32.644637 25336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=35781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:32.646158 25336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:32.647554 25336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:32.670990 25342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:32.671060 25343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:32.671070 25345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:32.675835 25336 server_base.cc:1034] running on GCE node
I20260924 01:06:32.676793 25336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:32.678542 25336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:32.679800 25336 hybrid_clock.cc:648] HybridClock initialized: now 1790211992679712 us; error 50 us; skew 500 ppm
I20260924 01:06:32.680194 25336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:32 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211992, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:32.691692 25336 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:06:32.693647 25336 webserver.cc:460] Webserver started at http://127.8.254.126:35781/ using document root <none> and password file <none>
I20260924 01:06:32.694581 25336 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:32.694734 25336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:32.700886 25336 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 01:06:32.704088 25352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:32.705591 25336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:06:32.705780 25336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "db7a530b79cd42eebacd195d0d0931c5"
format_stamp: "Formatted at 2026-09-24 01:06:09 on dist-test-slave-5c7l"
server_key: "98afc5af56b4ac4ec15b6e996821fc00"
server_key_iv: "b0bc22cdc6e824ba83159cb2e394b45d"
server_key_version: "encryptionkey@0"
I20260924 01:06:32.706988 25336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:32.723846 25336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:32.728261 25336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:32.728559 25336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:32.748122 25336 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39029
I20260924 01:06:32.748167 25403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39029 every 8 connection(s)
I20260924 01:06:32.749830 25336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:06:32.754324 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25336
I20260924 01:06:32.755391 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:34787
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:39029
--webserver_port=40765
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:06:32.759068 25404 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:06:32.764582 25404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap starting.
I20260924 01:06:32.785369 25404 log.cc:824] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:32.797914 25404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:06:32.798523 25404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap complete.
I20260924 01:06:32.808199 25404 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:32.809093 25404 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7a530b79cd42eebacd195d0d0931c5, State: Initialized, Role: FOLLOWER
I20260924 01:06:32.809784 25404 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:32.810022 25404 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:06:32.810148 25404 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:06:32.810403 25404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 01:06:32.814706 25404 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:32.815246 25404 leader_election.cc:302] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7a530b79cd42eebacd195d0d0931c5; no voters:
I20260924 01:06:32.816004 25404 leader_election.cc:288] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 01:06:32.816265 25408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 01:06:32.817498 25408 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 LEADER]: Becoming Leader. State: Replica: db7a530b79cd42eebacd195d0d0931c5, State: Running, Role: LEADER
I20260924 01:06:32.818138 25408 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:32.818603 25404 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:06:32.820811 25410 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7a530b79cd42eebacd195d0d0931c5. Latest consensus state: current_term: 2 leader_uuid: "db7a530b79cd42eebacd195d0d0931c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } } }
I20260924 01:06:32.821270 25410 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: This master's current role is: LEADER
I20260924 01:06:32.821792 25409 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db7a530b79cd42eebacd195d0d0931c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } } }
I20260924 01:06:32.822140 25409 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: This master's current role is: LEADER
I20260924 01:06:32.825985 25414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:06:32.834311 25414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:06:32.835770 25414 catalog_manager.cc:1259] Loaded cluster ID: 5557d5c986d9479daad155c6507b1f2e
I20260924 01:06:32.835934 25414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:06:32.840039 25414 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:06:32.841710 25414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Loaded TSK: 0
I20260924 01:06:32.880384 25414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Generated new TSK 1
I20260924 01:06:32.880817 25414 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 01:06:33.115837 25406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:33.116145 25406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:33.116303 25406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:33.123625 25406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:33.123775 25406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:33.123939 25406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:06:33.134405 25406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=40765
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:33.135732 25406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:33.137091 25406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:33.152812 25431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:33.152736 25432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:33.152994 25434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:33.154871 25406 server_base.cc:1034] running on GCE node
I20260924 01:06:33.155767 25406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:33.157146 25406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:33.158455 25406 hybrid_clock.cc:648] HybridClock initialized: now 1790211993158342 us; error 91 us; skew 500 ppm
I20260924 01:06:33.158888 25406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:33 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211993, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:33.169921 25406 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:06:33.172151 25406 webserver.cc:460] Webserver started at http://127.8.254.65:40765/ using document root <none> and password file <none>
I20260924 01:06:33.173265 25406 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:33.173452 25406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:33.181200 25406 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:06:33.185134 25441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:33.186738 25406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 01:06:33.186923 25406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "315abe3fc0c740779cf2dc36359464ef"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "b2e1a9859053377955aaac9275367e1f"
server_key_iv: "e0b069b1fcbfd6539878d7e5aaa09393"
server_key_version: "encryptionkey@0"
I20260924 01:06:33.187915 25406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:33.212810 25406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:33.217422 25406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:33.217715 25406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:33.218914 25406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:33.220371 25406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:33.220494 25406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:33.220630 25406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:33.220714 25406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:33.254714 25406 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:34787
I20260924 01:06:33.254801 25553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:34787 every 8 connection(s)
I20260924 01:06:33.256214 25406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:06:33.263586 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25406
I20260924 01:06:33.264470 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:35423
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:39029
--webserver_port=42861
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:06:33 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211993, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:33.290170 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:33.259581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:42635 (local address 127.8.254.126:39029)
0924 01:06:33.260501 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:33.260563 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:33.261554 (+ 991us) server_negotiation.cc:407] Connection header received
0924 01:06:33.265025 (+ 3471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:33.265050 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:33.266032 (+ 982us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:33.266751 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:33.268934 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.271527 (+ 2593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.273563 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.274779 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.275000 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:33.280040 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:33.280053 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:33.280076 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:33.280129 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:33.283863 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:33.285005 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:33.285015 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:33.285023 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:33.285266 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:33.285872 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:33.285879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:33.285884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:33.286696 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:33.286943 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:33.288233 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0924 01:06:33.288721 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":321,"threads_started":1}
I20260924 01:06:33.290263 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:33.259803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:42635)
0924 01:06:33.261335 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:33.261403 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:33.264006 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:33.267004 (+ 2998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:33.267017 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:33.267855 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:06:33.268698 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:33.268721 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.271784 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.271791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:33.273392 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:33.273406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.275073 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.275085 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:33.275455 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:33.276327 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:06:33.276363 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:06:33.279693 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:06:33.284056 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:33.284065 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:33.284082 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:33.284821 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:33.285432 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:33.285440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:33.285444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:33.285739 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:33.286924 (+ 1185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:06:33.286936 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:06:33.287638 (+ 702us) client_negotiation.cc:769] Sending connection context
0924 01:06:33.287974 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 01:06:33.288616 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":938,"threads_started":1}
I20260924 01:06:33.294961 25554 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39029
I20260924 01:06:33.295360 25554 heartbeater.cc:464] Registering TS with master...
I20260924 01:06:33.296386 25554 heartbeater.cc:511] Master 127.8.254.126:39029 requested a full tablet report, sending...
I20260924 01:06:33.299219 25369 ts_manager.cc:194] Registered new tserver with Master: 315abe3fc0c740779cf2dc36359464ef (127.8.254.65:34787)
I20260924 01:06:33.301276 25369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:42635
W20260924 01:06:33.616304 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:33.616611 25558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:33.616707 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:33.624027 25558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:33.624151 25558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:33.624269 25558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:06:33.634137 25558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:35423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=42861
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:33.635535 25558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:33.636839 25558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:33.650671 25563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:33.650671 25564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:33.650699 25566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:33.651471 25558 server_base.cc:1034] running on GCE node
I20260924 01:06:33.652104 25558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:33.653069 25558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:33.654325 25558 hybrid_clock.cc:648] HybridClock initialized: now 1790211993654264 us; error 49 us; skew 500 ppm
I20260924 01:06:33.654675 25558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:33 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211993, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:33.657795 25558 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:06:33.659476 25558 webserver.cc:460] Webserver started at http://127.8.254.66:42861/ using document root <none> and password file <none>
I20260924 01:06:33.660142 25558 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:33.660261 25558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:33.665890 25558 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 01:06:33.668706 25573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:33.670054 25558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:06:33.670254 25558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "83e1014bc297e7418ba5476fe80ea272"
server_key_iv: "99b7eaa7eec140dcb94ef40707b9cad1"
server_key_version: "encryptionkey@0"
I20260924 01:06:33.671219 25558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:33.693691 25558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:33.697911 25558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:33.698138 25558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:33.699134 25558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:33.700413 25558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:33.700505 25558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:33.700585 25558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:33.700644 25558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:33.734098 25558 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:35423
I20260924 01:06:33.734149 25685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:35423 every 8 connection(s)
I20260924 01:06:33.735591 25558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:06:33.743557 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25558
I20260924 01:06:33.744273 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:42615
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:39029
--webserver_port=45345
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 01:06:33 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211993, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:33.759138 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:33.738256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:55135 (local address 127.8.254.126:39029)
0924 01:06:33.738447 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:33.738452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:33.739613 (+ 1161us) server_negotiation.cc:407] Connection header received
0924 01:06:33.741389 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:33.741393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:33.741463 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:33.741552 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:33.743098 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.744212 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.745882 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.746672 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.746828 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:33.751454 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:33.751462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:33.751467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:33.751516 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:33.755206 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:33.756234 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:33.756245 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:33.756251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:33.756399 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:33.756927 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:33.756932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:33.756935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:33.757292 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:33.757447 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:33.758507 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 01:06:33.758767 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:06:33.760465 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:33.738765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:55135)
0924 01:06:33.739453 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:33.739495 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:33.741143 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:33.741768 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:33.741775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:33.742357 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:06:33.742927 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:33.742945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.744388 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.744395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:33.745717 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:33.745734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:33.746900 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:33.746906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:33.747104 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:33.747872 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:06:33.747911 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:06:33.751169 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:06:33.755374 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:33.755387 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:33.755407 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:33.756082 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:33.756518 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:33.756526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:33.756532 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:33.756796 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:33.757437 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:06:33.757448 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:06:33.757948 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 01:06:33.758304 (+ 356us) client_negotiation.cc:240] Negotiation successful
0924 01:06:33.758919 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
I20260924 01:06:33.762130 25686 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39029
I20260924 01:06:33.762550 25686 heartbeater.cc:464] Registering TS with master...
I20260924 01:06:33.763473 25686 heartbeater.cc:511] Master 127.8.254.126:39029 requested a full tablet report, sending...
I20260924 01:06:33.765144 25369 ts_manager.cc:194] Registered new tserver with Master: e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66:35423)
I20260924 01:06:33.766355 25369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:55135
W20260924 01:06:34.083285 25689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:34.083602 25689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:34.083776 25689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:34.091019 25689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:34.091179 25689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:34.091358 25689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:06:34.101300 25689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:42615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=45345
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:34.102638 25689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:34.104008 25689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:34.117072 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:34.117151 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:34.117413 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:34.118522 25689 server_base.cc:1034] running on GCE node
I20260924 01:06:34.119202 25689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:34.120154 25689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:34.121378 25689 hybrid_clock.cc:648] HybridClock initialized: now 1790211994121322 us; error 64 us; skew 500 ppm
I20260924 01:06:34.121687 25689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:34 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:34.124857 25689 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:06:34.126346 25689 webserver.cc:460] Webserver started at http://127.8.254.67:45345/ using document root <none> and password file <none>
I20260924 01:06:34.127002 25689 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:34.127117 25689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:34.132484 25689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 01:06:34.135229 25704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:34.136520 25689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:06:34.136655 25689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "2507eb95da124322a0fe41d4cce79ef8"
format_stamp: "Formatted at 2026-09-24 01:06:11 on dist-test-slave-5c7l"
server_key: "bb2cd1c28b66c1677da195981847b817"
server_key_iv: "7feeebf9126bf38b183db62dcfd36a90"
server_key_version: "encryptionkey@0"
I20260924 01:06:34.137586 25689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:34.155016 25689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:34.159714 25689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:34.159958 25689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:34.160966 25689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:34.162305 25689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:34.162422 25689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:34.162560 25689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:34.162652 25689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:34.195645 25689 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:42615
I20260924 01:06:34.195731 25816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:42615 every 8 connection(s)
I20260924 01:06:34.197279 25689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:06:34.202716 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25689
Sep 24 01:06:34 dist-test-slave-5c7l krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:06:34.218782 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.199802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:36077 (local address 127.8.254.126:39029)
0924 01:06:34.199979 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.199984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.201047 (+ 1063us) server_negotiation.cc:407] Connection header received
0924 01:06:34.203215 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.203220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.203289 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.203408 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:34.205373 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.206573 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.208058 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.208705 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.208846 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.212663 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:34.212674 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:34.212677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:34.212708 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:34.215586 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.216307 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.216315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.216320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.216438 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.216848 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.216854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.216857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.217136 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:34.217412 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.218151 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.218412 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:06:34.219565 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.203809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.8.254.126:39029)
0924 01:06:34.204747 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.204751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.204764 (+ 13us) server_negotiation.cc:407] Connection header received
0924 01:06:34.204804 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.204807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.204858 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.205049 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:34.207102 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.207804 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.209633 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.210325 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.210480 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.213193 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:34.213197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:34.213200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:34.213273 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:34.216235 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.216884 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.216890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.216893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.217019 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.217403 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.217409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.217412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.217675 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:34.218052 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.218974 (+ 922us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.219196 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":731,"threads_started":1}
I20260924 01:06:34.219864 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.200204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:36077)
0924 01:06:34.200890 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.200935 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:34.202913 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:34.203639 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:34.203651 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:34.204522 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:06:34.205212 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.205231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.206727 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.206731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.207928 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.207940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.208909 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.208914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.209121 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.209817 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:06:34.209853 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:06:34.212466 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:06:34.215726 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:34.215735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:34.215753 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:34.216177 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:34.216547 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:06:34.216552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:06:34.216556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:06:34.216740 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:06:34.217373 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:06:34.217381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:06:34.217794 (+ 413us) client_negotiation.cc:769] Sending connection context
0924 01:06:34.218077 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 01:06:34.218649 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":266,"threads_started":1}
I20260924 01:06:34.221359 25817 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39029
I20260924 01:06:34.221778 25817 heartbeater.cc:464] Registering TS with master...
I20260924 01:06:34.222599 25817 heartbeater.cc:511] Master 127.8.254.126:39029 requested a full tablet report, sending...
I20260924 01:06:34.224016 25369 ts_manager.cc:194] Registered new tserver with Master: 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615)
I20260924 01:06:34.224812 25369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:36077
I20260924 01:06:34.234732 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:06:34.241324 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.235631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.8.254.126:39029)
0924 01:06:34.235896 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.235904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.236213 (+ 309us) server_negotiation.cc:407] Connection header received
0924 01:06:34.236408 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.236412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.236551 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.236674 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:06:34.237130 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.237852 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.239437 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.239910 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.240004 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.240449 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:06:34.240703 (+ 254us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 01:06:34.241025 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40222: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 01:06:34.241462 25821 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40222: token expired
W20260924 01:06:34.241987 25290 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.254.126:39029)
I20260924 01:06:34.254945 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.242465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.8.254.126:39029)
0924 01:06:34.242628 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.242632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.242647 (+ 15us) server_negotiation.cc:407] Connection header received
0924 01:06:34.242799 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.242803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.242857 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.242932 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:34.244394 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.245351 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.246712 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.247102 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.247195 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.249716 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:34.249720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:34.249723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:34.249754 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:34.252305 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.253065 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.253069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.253071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.253201 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.253548 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.253551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.253553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.253868 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:34.254044 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.254511 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.254680 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:06:34.259001 25369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:06:34.266938 25369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:06:34.290441 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.279997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:35423 (local address 127.0.0.1:52562)
0924 01:06:34.282014 (+ 2017us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.282032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:34.282163 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:34.283181 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:34.283195 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:34.283221 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:34.283618 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.283637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.285750 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.285756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.287836 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.287849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.289310 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.289317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.289531 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.289591 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 01:06:34.289734 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 01:06:34.289955 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"mutex_wait_us":127,"thread_start_us":1771,"threads_started":1}
I20260924 01:06:34.291280 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.280235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.8.254.66:35423)
0924 01:06:34.280925 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.280938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.282179 (+ 1241us) server_negotiation.cc:407] Connection header received
0924 01:06:34.282393 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.282406 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.282702 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.282941 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:34.283890 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.285566 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.287998 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.289046 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.289235 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.289464 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.290340 (+ 876us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.290971 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":983,"threads_started":1}
I20260924 01:06:34.292095 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.280326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:42615 (local address 127.0.0.1:53486)
0924 01:06:34.282636 (+ 2310us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.282656 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:34.282787 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:34.283463 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:34.283472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:34.283499 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:34.283877 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.283892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.286187 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.286191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.288821 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.288838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.291092 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.291099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.291321 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.291358 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 01:06:34.291443 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 01:06:34.291598 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2196,"mutex_wait_us":115,"thread_start_us":2254,"threads_started":1}
I20260924 01:06:34.292145 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.280944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.8.254.67:42615)
0924 01:06:34.281802 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.281814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.282921 (+ 1107us) server_negotiation.cc:407] Connection header received
0924 01:06:34.283026 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.283034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.283273 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.283457 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:34.284250 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.285981 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.289002 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.290505 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.290692 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.290939 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.291635 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.291816 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":466,"threads_started":1}
I20260924 01:06:34.292130 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.278910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49524)
0924 01:06:34.279745 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.279798 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:34.280018 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:34.283795 (+ 3777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:34.283800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:34.283830 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:34.284267 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.284280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.286103 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.286114 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.288834 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.288844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.291368 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.291377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.291556 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.291594 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 01:06:34.291688 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 01:06:34.291826 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":125,"thread_start_us":353,"threads_started":1}
I20260924 01:06:34.292886 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.279235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.8.254.65:34787)
0924 01:06:34.283159 (+ 3924us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.283168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.283192 (+ 24us) server_negotiation.cc:407] Connection header received
0924 01:06:34.283294 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.283305 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.283534 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.283795 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:34.284451 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.285960 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.289576 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.291088 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.291269 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.291522 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.292383 (+ 861us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.292573 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3758,"thread_start_us":3613,"threads_started":1}
I20260924 01:06:34.296011 25621 tablet_service.cc:1467] Processing CreateTablet for tablet 093556ff717e4de29ef98332701310db (DEFAULT_TABLE table=test-table [id=26c2eab202b44e7b8da40a57a648ff0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:06:34.296237 25752 tablet_service.cc:1467] Processing CreateTablet for tablet 093556ff717e4de29ef98332701310db (DEFAULT_TABLE table=test-table [id=26c2eab202b44e7b8da40a57a648ff0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:06:34.296595 25489 tablet_service.cc:1467] Processing CreateTablet for tablet 093556ff717e4de29ef98332701310db (DEFAULT_TABLE table=test-table [id=26c2eab202b44e7b8da40a57a648ff0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:06:34.297349 25621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093556ff717e4de29ef98332701310db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:34.297667 25752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093556ff717e4de29ef98332701310db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:34.297924 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093556ff717e4de29ef98332701310db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:06:34.305843 25554 heartbeater.cc:503] Master 127.8.254.126:39029 was elected leader, sending a full tablet report...
I20260924 01:06:34.310161 25833 tablet_bootstrap.cc:492] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Bootstrap starting.
I20260924 01:06:34.313482 25834 tablet_bootstrap.cc:492] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Bootstrap starting.
I20260924 01:06:34.314667 25833 tablet_bootstrap.cc:654] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:34.316197 25833 log.cc:824] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:34.316532 25835 tablet_bootstrap.cc:492] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Bootstrap starting.
I20260924 01:06:34.317076 25834 tablet_bootstrap.cc:654] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:34.318195 25834 log.cc:824] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:34.319959 25835 tablet_bootstrap.cc:654] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Neither blocks nor log segments found. Creating new log.
I20260924 01:06:34.320142 25833 tablet_bootstrap.cc:492] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: No bootstrap required, opened a new log
I20260924 01:06:34.320425 25833 ts_tablet_manager.cc:1397] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260924 01:06:34.320957 25835 log.cc:824] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:34.321385 25834 tablet_bootstrap.cc:492] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: No bootstrap required, opened a new log
I20260924 01:06:34.321647 25834 ts_tablet_manager.cc:1397] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 01:06:34.324193 25835 tablet_bootstrap.cc:492] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: No bootstrap required, opened a new log
I20260924 01:06:34.324446 25835 ts_tablet_manager.cc:1397] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 01:06:34.331595 25835 raft_consensus.cc:348] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.331595 25833 raft_consensus.cc:348] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.332078 25835 raft_consensus.cc:374] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:34.332077 25833 raft_consensus.cc:374] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:34.332176 25835 raft_consensus.cc:729] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2507eb95da124322a0fe41d4cce79ef8, State: Initialized, Role: FOLLOWER
I20260924 01:06:34.332185 25833 raft_consensus.cc:729] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d3ba02424340fd9e2a70f5b1aa6aa3, State: Initialized, Role: FOLLOWER
I20260924 01:06:34.332255 25834 raft_consensus.cc:348] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.332666 25834 raft_consensus.cc:374] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:06:34.332768 25834 raft_consensus.cc:729] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315abe3fc0c740779cf2dc36359464ef, State: Initialized, Role: FOLLOWER
I20260924 01:06:34.332849 25835 consensus_queue.cc:260] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.332901 25833 consensus_queue.cc:260] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.333315 25834 consensus_queue.cc:260] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.334125 25817 heartbeater.cc:503] Master 127.8.254.126:39029 was elected leader, sending a full tablet report...
I20260924 01:06:34.334813 25835 ts_tablet_manager.cc:1428] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 01:06:34.336207 25686 heartbeater.cc:503] Master 127.8.254.126:39029 was elected leader, sending a full tablet report...
I20260924 01:06:34.337117 25833 ts_tablet_manager.cc:1428] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260924 01:06:34.339028 25834 ts_tablet_manager.cc:1428] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260924 01:06:34.349715 25841 raft_consensus.cc:482] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:06:34.350036 25841 raft_consensus.cc:504] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.351788 25841 leader_election.cc:288] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66:35423), 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615)
I20260924 01:06:34.361409 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.352333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:40205 (local address 127.8.254.67:42615)
0924 01:06:34.352661 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.352670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.353473 (+ 803us) server_negotiation.cc:407] Connection header received
0924 01:06:34.353688 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.353693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.353771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.353878 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:34.354819 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.355794 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.358868 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.360157 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.360322 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.360419 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.360979 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.361119 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 01:06:34.361398 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.352310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:41087 (local address 127.8.254.66:35423)
0924 01:06:34.352535 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.352539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.353474 (+ 935us) server_negotiation.cc:407] Connection header received
0924 01:06:34.353688 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.353693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.353771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.353877 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:06:34.354758 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.355665 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.358887 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.360135 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.360312 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.360392 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.360979 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.361119 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 01:06:34.361732 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.352246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:42615 (local address 127.8.254.65:40205)
0924 01:06:34.353367 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.353385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:34.353536 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:34.353913 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:34.353918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:34.353952 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:34.354571 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.354591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.355916 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.355926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.358692 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.358710 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.360445 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.360451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.360759 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.360836 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 01:06:34.360938 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 01:06:34.361206 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":304,"spinlock_wait_cycles":17664,"thread_start_us":817,"threads_started":1}
I20260924 01:06:34.361755 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.352142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:35423 (local address 127.8.254.65:41087)
0924 01:06:34.353320 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.353350 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 01:06:34.353536 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:06:34.353913 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:06:34.353918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:06:34.353954 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:06:34.354598 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.354608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.355868 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.355873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.358699 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:06:34.358710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.360456 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.360461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:06:34.360762 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.360839 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 01:06:34.360938 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 01:06:34.361206 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":424,"spinlock_wait_cycles":25984,"thread_start_us":722,"threads_started":1}
I20260924 01:06:34.362884 25772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093556ff717e4de29ef98332701310db" candidate_uuid: "315abe3fc0c740779cf2dc36359464ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2507eb95da124322a0fe41d4cce79ef8" is_pre_election: true
I20260924 01:06:34.362893 25641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093556ff717e4de29ef98332701310db" candidate_uuid: "315abe3fc0c740779cf2dc36359464ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" is_pre_election: true
I20260924 01:06:34.363407 25772 raft_consensus.cc:2454] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315abe3fc0c740779cf2dc36359464ef in term 0.
I20260924 01:06:34.363409 25641 raft_consensus.cc:2454] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315abe3fc0c740779cf2dc36359464ef in term 0.
I20260924 01:06:34.364107 25445 leader_election.cc:302] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2507eb95da124322a0fe41d4cce79ef8, 315abe3fc0c740779cf2dc36359464ef, e8d3ba02424340fd9e2a70f5b1aa6aa3; no voters:
I20260924 01:06:34.364562 25841 raft_consensus.cc:2781] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:06:34.364732 25841 raft_consensus.cc:482] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:06:34.364841 25841 raft_consensus.cc:3037] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:34.367197 25841 raft_consensus.cc:504] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.367797 25841 leader_election.cc:288] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [CANDIDATE]: Term 1 election: Requested vote from peers e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66:35423), 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615)
I20260924 01:06:34.368323 25641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093556ff717e4de29ef98332701310db" candidate_uuid: "315abe3fc0c740779cf2dc36359464ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3"
I20260924 01:06:34.368340 25772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093556ff717e4de29ef98332701310db" candidate_uuid: "315abe3fc0c740779cf2dc36359464ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2507eb95da124322a0fe41d4cce79ef8"
I20260924 01:06:34.368530 25641 raft_consensus.cc:3037] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:34.368541 25772 raft_consensus.cc:3037] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:06:34.371654 25772 raft_consensus.cc:2454] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315abe3fc0c740779cf2dc36359464ef in term 1.
I20260924 01:06:34.372102 25445 leader_election.cc:302] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2507eb95da124322a0fe41d4cce79ef8, 315abe3fc0c740779cf2dc36359464ef; no voters:
I20260924 01:06:34.372128 25641 raft_consensus.cc:2454] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315abe3fc0c740779cf2dc36359464ef in term 1.
I20260924 01:06:34.372474 25841 raft_consensus.cc:2781] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:06:34.372985 25841 raft_consensus.cc:686] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 1 LEADER]: Becoming Leader. State: Replica: 315abe3fc0c740779cf2dc36359464ef, State: Running, Role: LEADER
I20260924 01:06:34.373589 25841 consensus_queue.cc:237] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } }
I20260924 01:06:34.378386 25367 catalog_manager.cc:5495] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 315abe3fc0c740779cf2dc36359464ef (127.8.254.65). New cstate: current_term: 1 leader_uuid: "315abe3fc0c740779cf2dc36359464ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315abe3fc0c740779cf2dc36359464ef" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 } health_report { overall_health: UNKNOWN } } }
I20260924 01:06:34.404950 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.399600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.8.254.65:34787)
0924 01:06:34.399818 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.399825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.399844 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:06:34.400058 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.400062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.400149 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.400318 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:06:34.400670 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.401401 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.402814 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.403216 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.403354 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.403861 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:06:34.404360 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:06:34.404515 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.404607 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.404685 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:06:34.412995 25772 raft_consensus.cc:1261] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Refusing update from remote peer 315abe3fc0c740779cf2dc36359464ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:06:34.412995 25641 raft_consensus.cc:1261] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Refusing update from remote peer 315abe3fc0c740779cf2dc36359464ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:06:34.413978 25841 consensus_queue.cc:1035] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2507eb95da124322a0fe41d4cce79ef8" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 42615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:06:34.414290 25844 consensus_queue.cc:1035] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:06:34.425781 25847 mvcc.cc:205] Tried to move back new op lower bound from 7332708329106075648 to 7332708328958746624. Current Snapshot: MvccSnapshot[applied={T|T < 7332708329106075648}]
I20260924 01:06:34.439944 25367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:06:34.440249 25367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:06:34.443979 25367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Sending DeleteTablet for 3 replicas of tablet 093556ff717e4de29ef98332701310db
I20260924 01:06:34.445086 25489 tablet_service.cc:1514] Processing DeleteTablet for tablet 093556ff717e4de29ef98332701310db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:06:34 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49524
I20260924 01:06:34.445230 25621 tablet_service.cc:1514] Processing DeleteTablet for tablet 093556ff717e4de29ef98332701310db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:06:34 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:52562
I20260924 01:06:34.445438 25752 tablet_service.cc:1514] Processing DeleteTablet for tablet 093556ff717e4de29ef98332701310db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:06:34 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:53486
I20260924 01:06:34.446231 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25406
I20260924 01:06:34.459216 25857 tablet_replica.cc:330] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: stopping tablet replica
I20260924 01:06:34.459896 25857 raft_consensus.cc:2229] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:06:34.460888 25857 raft_consensus.cc:2258] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:06:34.463806 25857 ts_tablet_manager.cc:1905] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:34.467859 25859 tablet_replica.cc:330] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: stopping tablet replica
I20260924 01:06:34.468333 25859 raft_consensus.cc:2229] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:06:34.468894 25859 raft_consensus.cc:2258] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:06:34.471496 25859 ts_tablet_manager.cc:1905] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:34.474534 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.447702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.8.254.65:34787)
0924 01:06:34.447869 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.447875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.447893 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:06:34.447951 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.447956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.448029 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.448149 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:34.450290 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.451584 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.453586 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.454364 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.454499 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.462267 (+ 7768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:34.462285 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:34.462304 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:34.462366 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:34.468491 (+ 6125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.469803 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.469815 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.469820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.470023 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.471348 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.471356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.471359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.471848 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:34.472168 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.473880 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.474143 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:06:34.475657 25549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:34.477346 25858 tablet_replica.cc:330] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: stopping tablet replica
I20260924 01:06:34.478012 25858 raft_consensus.cc:2229] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:06:34.478825 25858 raft_consensus.cc:2258] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:06:34.481190 25858 ts_tablet_manager.cc:1905] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:34.481837 25859 ts_tablet_manager.cc:1918] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:06:34.482017 25859 log.cc:1196] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/093556ff717e4de29ef98332701310db
I20260924 01:06:34.482800 25859 ts_tablet_manager.cc:1939] T 093556ff717e4de29ef98332701310db P 2507eb95da124322a0fe41d4cce79ef8: Deleting consensus metadata
I20260924 01:06:34.484997 25356 catalog_manager.cc:4841] TS 2507eb95da124322a0fe41d4cce79ef8 (127.8.254.67:42615): tablet 093556ff717e4de29ef98332701310db (table test-table [id=26c2eab202b44e7b8da40a57a648ff0b]) successfully deleted
I20260924 01:06:34.487229 25858 ts_tablet_manager.cc:1918] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:06:34.487406 25858 log.cc:1196] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/093556ff717e4de29ef98332701310db
I20260924 01:06:34.487856 25858 ts_tablet_manager.cc:1939] T 093556ff717e4de29ef98332701310db P e8d3ba02424340fd9e2a70f5b1aa6aa3: Deleting consensus metadata
I20260924 01:06:34.489336 25353 catalog_manager.cc:4841] TS e8d3ba02424340fd9e2a70f5b1aa6aa3 (127.8.254.66:35423): tablet 093556ff717e4de29ef98332701310db (table test-table [id=26c2eab202b44e7b8da40a57a648ff0b]) successfully deleted
I20260924 01:06:34.577908 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25406
W20260924 01:06:34.597904 25353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.254.65:34787 (error 108)
I20260924 01:06:34.598043 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25558
I20260924 01:06:34.600097 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.599608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49544)
0924 01:06:34.599808 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.599926 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 01:06:34.600395 25353 catalog_manager.cc:4584] TS 315abe3fc0c740779cf2dc36359464ef (127.8.254.65:34787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 093556ff717e4de29ef98332701310db: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
I20260924 01:06:34.613310 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.599261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.8.254.66:35423)
0924 01:06:34.599433 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.599437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.599455 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:06:34.599628 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.599632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.599770 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.599891 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:34.601856 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.602755 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.604159 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.604558 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.604761 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.607586 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:34.607601 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:34.607617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:34.607694 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:34.610585 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.611340 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.611364 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.611370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.611522 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.611852 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.611856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.611858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.612283 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:34.612533 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.612787 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.612949 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:06:34.614050 25681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:34.654397 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.653808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49552)
0924 01:06:34.654044 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.654162 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 01:06:34.655421 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.655070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49566)
0924 01:06:34.655216 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.655307 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 01:06:34.705310 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25558
I20260924 01:06:34.716260 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.715613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49572)
0924 01:06:34.715899 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.716024 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 01:06:34.717439 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.716981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49574)
0924 01:06:34.717153 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.717275 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 01:06:34.723819 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25689
I20260924 01:06:34.736905 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.724530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.8.254.67:42615)
0924 01:06:34.724684 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 01:06:34.724688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:06:34.724707 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:06:34.724862 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:06:34.724867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:06:34.724932 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:06:34.725017 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:06:34.726505 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.727276 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.728666 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:06:34.729036 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:06:34.729126 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:06:34.731815 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:06:34.731830 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:06:34.731847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:06:34.731883 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:06:34.734530 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.735191 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.735199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.735201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.735360 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:06:34.735679 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:06:34.735682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:06:34.735684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:06:34.736022 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:06:34.736245 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 01:06:34.736495 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 01:06:34.736664 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 01:06:34.737643 25812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:34.797827 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.797202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49584)
0924 01:06:34.797459 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.797575 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 01:06:34.799008 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:34.798563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34787 (local address 127.0.0.1:49588)
0924 01:06:34.798777 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:34.798877 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 01:06:34.827142 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25689
I20260924 01:06:34.844300 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25336
I20260924 01:06:34.845448 25399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:06:34.961752 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25336
I20260924 01:06:54.984615 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39029
--webserver_interface=127.8.254.126
--webserver_port=35781
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:06:55.332286 25865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:55.332686 25865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:55.332855 25865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:55.340521 25865 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 01:06:55.340683 25865 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 01:06:55.340783 25865 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 01:06:55.340878 25865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:06:55.340960 25865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:55.341058 25865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:55.341145 25865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:06:55.341234 25865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:06:55.351464 25865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=35781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:55.352847 25865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:55.354245 25865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 01:06:55.366954 25865 server_base.cc:1034] running on GCE node
W20260924 01:06:55.366833 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:55.366853 25870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:55.366796 25871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:55.368201 25865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:55.369815 25865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:55.371073 25865 hybrid_clock.cc:648] HybridClock initialized: now 1790212015371011 us; error 45 us; skew 500 ppm
I20260924 01:06:55.371434 25865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:55 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:55.374806 25865 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:06:55.376343 25865 webserver.cc:460] Webserver started at http://127.8.254.126:35781/ using document root <none> and password file <none>
I20260924 01:06:55.377084 25865 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:55.377224 25865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:55.382915 25865 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 01:06:55.385907 25880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:55.387373 25865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:06:55.387573 25865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "db7a530b79cd42eebacd195d0d0931c5"
format_stamp: "Formatted at 2026-09-24 01:06:09 on dist-test-slave-5c7l"
server_key: "98afc5af56b4ac4ec15b6e996821fc00"
server_key_iv: "b0bc22cdc6e824ba83159cb2e394b45d"
server_key_version: "encryptionkey@0"
I20260924 01:06:55.388573 25865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:55.407167 25865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:55.411242 25865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:55.411551 25865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:55.430930 25865 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39029
I20260924 01:06:55.430954 25931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39029 every 8 connection(s)
I20260924 01:06:55.432495 25865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:06:55.439574 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25865
I20260924 01:06:55.439873 25932 sys_catalog.cc:263] Verifying existing consensus state
I20260924 01:06:55.440693 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:34787
--local_ip_for_outbound_sockets=127.8.254.65
--tserver_master_addrs=127.8.254.126:39029
--webserver_port=40765
--webserver_interface=127.8.254.65
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:06:55.445267 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap starting.
I20260924 01:06:55.467602 25932 log.cc:824] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:06:55.484395 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 01:06:55.484967 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Bootstrap complete.
I20260924 01:06:55.498891 25932 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:55.499830 25932 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7a530b79cd42eebacd195d0d0931c5, State: Initialized, Role: FOLLOWER
I20260924 01:06:55.500577 25932 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:55.500840 25932 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:06:55.500958 25932 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:06:55.501154 25932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 2 FOLLOWER]: Advancing to term 3
I20260924 01:06:55.505381 25932 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:55.505892 25932 leader_election.cc:302] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7a530b79cd42eebacd195d0d0931c5; no voters:
I20260924 01:06:55.506681 25932 leader_election.cc:288] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 01:06:55.506834 25936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 01:06:55.509145 25932 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:06:55.513965 25936 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [term 3 LEADER]: Becoming Leader. State: Replica: db7a530b79cd42eebacd195d0d0931c5, State: Running, Role: LEADER
I20260924 01:06:55.514745 25936 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } }
I20260924 01:06:55.517563 25937 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "db7a530b79cd42eebacd195d0d0931c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } } }
I20260924 01:06:55.517998 25937 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: This master's current role is: LEADER
I20260924 01:06:55.518676 25938 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7a530b79cd42eebacd195d0d0931c5. Latest consensus state: current_term: 3 leader_uuid: "db7a530b79cd42eebacd195d0d0931c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7a530b79cd42eebacd195d0d0931c5" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39029 } } }
I20260924 01:06:55.519030 25938 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5 [sys.catalog]: This master's current role is: LEADER
I20260924 01:06:55.520994 25947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:06:55.536211 25947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:06:55.538273 25947 catalog_manager.cc:1259] Loaded cluster ID: 5557d5c986d9479daad155c6507b1f2e
I20260924 01:06:55.541020 25947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:06:55.546482 25947 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:06:55.548674 25947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Loaded TSK: 0
I20260924 01:06:55.548846 25947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db7a530b79cd42eebacd195d0d0931c5: Loaded TSK: 1
F20260924 01:06:55.550350 25947 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe81a0b89ad google::LogMessage::Fail() at ??:0
@ 0x7fe81a0bb88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe81a0b84de google::LogMessage::Flush() at ??:0
@ 0x7fe81a0b9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe81de1c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe81de0b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe82fae71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe82fae6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe82fb722a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe82fb720c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe82fbbc5e9 std::function<>::operator()() at ??:0
@ 0x7fe82faeacdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe82fae8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe82fb70610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe82fb70439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe81cd04345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe81cd0791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe81cd07799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe81cce2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe81ffbc6db start_thread at ??:0
@ 0x7fe818f5261f clone at ??:0
W20260924 01:06:55.832381 25934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:55.832721 25934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:55.832886 25934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:55.840754 25934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:55.840916 25934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:55.841084 25934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:06:55.851264 25934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=40765
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:55.852675 25934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:55.854523 25934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:55.868806 25981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:55.868741 25980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:55.868809 25983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:55.869664 25934 server_base.cc:1034] running on GCE node
I20260924 01:06:55.870385 25934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:55.871374 25934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:55.872611 25934 hybrid_clock.cc:648] HybridClock initialized: now 1790212015872561 us; error 61 us; skew 500 ppm
I20260924 01:06:55.872943 25934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:55 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:55.876250 25934 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:06:55.877830 25934 webserver.cc:460] Webserver started at http://127.8.254.65:40765/ using document root <none> and password file <none>
I20260924 01:06:55.878610 25934 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:55.878770 25934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:55.884121 25934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 01:06:55.887230 25990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:55.888608 25934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 01:06:55.888792 25934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "315abe3fc0c740779cf2dc36359464ef"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "b2e1a9859053377955aaac9275367e1f"
server_key_iv: "e0b069b1fcbfd6539878d7e5aaa09393"
server_key_version: "encryptionkey@0"
I20260924 01:06:55.889784 25934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:55.905153 25934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:55.909138 25934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:55.909449 25934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:55.910530 25934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:55.912582 25997 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 01:06:55.918406 25997 ts_tablet_manager.cc:1852] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 01:06:55.919464 25997 ts_tablet_manager.cc:1905] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:06:55.927290 25997 ts_tablet_manager.cc:1918] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:06:55.927477 25997 log.cc:1196] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/093556ff717e4de29ef98332701310db
I20260924 01:06:55.928037 25997 ts_tablet_manager.cc:1939] T 093556ff717e4de29ef98332701310db P 315abe3fc0c740779cf2dc36359464ef: Deleting consensus metadata
I20260924 01:06:55.928427 25934 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 01:06:55.928582 25934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260924 01:06:55.928719 25934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:55.928809 25934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:55.964017 25934 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:34787
I20260924 01:06:55.964107 26103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:34787 every 8 connection(s)
I20260924 01:06:55.965416 25934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:06:55.969177 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 25934
I20260924 01:06:55.969982 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:35423
--local_ip_for_outbound_sockets=127.8.254.66
--tserver_master_addrs=127.8.254.126:39029
--webserver_port=42861
--webserver_interface=127.8.254.66
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:06:55.970743 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:55.968623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:55373)
0924 01:06:55.969495 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:55.969751 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":312,"threads_started":1}
W20260924 01:06:55.971378 26104 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:39029 (0 consecutive failures): Network error: Failed to ping master at 127.8.254.126:39029: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
I20260924 01:06:55.972254 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:55.971907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:33775)
0924 01:06:55.972072 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:55.972150 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 01:06:55.973205 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:55.972867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:60009)
0924 01:06:55.973014 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:55.973093 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 01:06:55.973443 26104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:06:56.309713 26107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:56.310058 26107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:56.310273 26107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:56.317852 26107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:56.317993 26107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:56.318171 26107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:06:56.328123 26107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:35423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=42861
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:56.329447 26107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:56.330883 26107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:56.345026 26113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:56.345090 26112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:56.345048 26115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:56.346179 26107 server_base.cc:1034] running on GCE node
I20260924 01:06:56.346976 26107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:56.348033 26107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:56.349303 26107 hybrid_clock.cc:648] HybridClock initialized: now 1790212016349227 us; error 67 us; skew 500 ppm
I20260924 01:06:56.349645 26107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:56 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212016, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:56.353101 26107 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:06:56.354835 26107 webserver.cc:460] Webserver started at http://127.8.254.66:42861/ using document root <none> and password file <none>
I20260924 01:06:56.355587 26107 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:56.355742 26107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:56.361438 26107 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 01:06:56.364493 26122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:56.365926 26107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:06:56.366115 26107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "e8d3ba02424340fd9e2a70f5b1aa6aa3"
format_stamp: "Formatted at 2026-09-24 01:06:10 on dist-test-slave-5c7l"
server_key: "83e1014bc297e7418ba5476fe80ea272"
server_key_iv: "99b7eaa7eec140dcb94ef40707b9cad1"
server_key_version: "encryptionkey@0"
I20260924 01:06:56.367229 26107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:56.379173 26107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:56.383142 26107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:56.383435 26107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:56.384409 26107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:56.385731 26107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:56.385852 26107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:56.385993 26107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:56.386083 26107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:56.420114 26107 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:35423
I20260924 01:06:56.420205 26234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:35423 every 8 connection(s)
I20260924 01:06:56.421664 26107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:06:56.424880 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26107
I20260924 01:06:56.425504 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:42615
--local_ip_for_outbound_sockets=127.8.254.67
--tserver_master_addrs=127.8.254.126:39029
--webserver_port=45345
--webserver_interface=127.8.254.67
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:06:56.426762 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.424607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:41519)
0924 01:06:56.425440 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.425738 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":346,"threads_started":1}
W20260924 01:06:56.427453 26235 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:39029 (0 consecutive failures): Network error: Failed to ping master at 127.8.254.126:39029: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
I20260924 01:06:56.428369 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.428050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:60397)
0924 01:06:56.428198 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.428267 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 01:06:56.429348 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.428983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:57905)
0924 01:06:56.429132 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.429217 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 01:06:56.429606 26235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 01:06:56.760918 26238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:06:56.761198 26238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:06:56.761415 26238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:06:56.768623 26238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:06:56.768734 26238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:06:56.768941 26238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:06:56.779135 26238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:42615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=45345
--tserver_master_addrs=127.8.254.126:39029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:06:56.780520 26238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:06:56.782001 26238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:06:56.795747 26246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:56.795773 26244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:06:56.795728 26243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:06:56.797364 26238 server_base.cc:1034] running on GCE node
I20260924 01:06:56.798069 26238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:06:56.799244 26238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:06:56.800490 26238 hybrid_clock.cc:648] HybridClock initialized: now 1790212016800419 us; error 64 us; skew 500 ppm
I20260924 01:06:56.800845 26238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:06:56 dist-test-slave-5c7l krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212016, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:06:56.804188 26238 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:06:56.805806 26238 webserver.cc:460] Webserver started at http://127.8.254.67:45345/ using document root <none> and password file <none>
I20260924 01:06:56.806612 26238 fs_manager.cc:359] Metadata directory not provided
I20260924 01:06:56.806773 26238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:06:56.812448 26238 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 01:06:56.815304 26253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:56.816632 26238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:06:56.816819 26238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "2507eb95da124322a0fe41d4cce79ef8"
format_stamp: "Formatted at 2026-09-24 01:06:11 on dist-test-slave-5c7l"
server_key: "bb2cd1c28b66c1677da195981847b817"
server_key_iv: "7feeebf9126bf38b183db62dcfd36a90"
server_key_version: "encryptionkey@0"
I20260924 01:06:56.817816 26238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:06:56.840902 26238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:06:56.845156 26238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:06:56.845467 26238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:06:56.846496 26238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:06:56.847863 26238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:06:56.848007 26238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:56.848143 26238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:06:56.848230 26238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:06:56.881744 26238 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:42615
I20260924 01:06:56.881834 26365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:42615 every 8 connection(s)
I20260924 01:06:56.883322 26238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:06:56.888084 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.886321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:34287)
0924 01:06:56.887023 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.887256 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
W20260924 01:06:56.888748 26366 heartbeater.cc:644] Failed to heartbeat to 127.8.254.126:39029 (0 consecutive failures): Network error: Failed to ping master at 127.8.254.126:39029: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
I20260924 01:06:56.889542 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.889223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:44453)
0924 01:06:56.889367 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.889432 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 01:06:56.890398 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.890043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:55809)
0924 01:06:56.890157 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.890288 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 01:06:56.890688 26366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 01:06:56.890702 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26238
I20260924 01:06:56.974932 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:56.974054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:40261)
0924 01:06:56.974588 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:56.974685 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":283,"threads_started":1}
I20260924 01:06:57.431063 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:57.430271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:40325)
0924 01:06:57.430722 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:57.430836 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260924 01:06:57.892305 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:57.891328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:55733)
0924 01:06:57.891914 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:57.892015 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":287,"threads_started":1}
I20260924 01:06:57.976509 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:57.975833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:56697)
0924 01:06:57.976269 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:57.976354 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260924 01:06:58.432680 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:58.431931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:59865)
0924 01:06:58.432436 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:58.432516 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":241,"threads_started":1}
I20260924 01:06:58.894243 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:58.893285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:52341)
0924 01:06:58.893936 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:58.894018 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":393,"threads_started":1}
I20260924 01:06:58.978439 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:58.977591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:56239)
0924 01:06:58.978063 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:58.978152 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":231,"threads_started":1}
I20260924 01:06:59.434420 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:59.433603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:44343)
0924 01:06:59.434096 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:59.434179 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260924 01:06:59.895836 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:59.895116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:59601)
0924 01:06:59.895588 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:59.895666 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
I20260924 01:06:59.980218 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:06:59.979428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:46379)
0924 01:06:59.979954 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 01:06:59.980048 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":257,"threads_started":1}
I20260924 01:07:00.436165 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:00.435379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:35027)
0924 01:07:00.435876 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:00.435962 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260924 01:07:00.897600 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:00.896841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:42249)
0924 01:07:00.897299 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:00.897399 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260924 01:07:00.982051 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:00.981322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:34607)
0924 01:07:00.981796 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:00.981882 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
I20260924 01:07:01.438218 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:01.437257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:57395)
0924 01:07:01.437721 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:01.437808 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
I20260924 01:07:01.899463 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:01.898670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:46669)
0924 01:07:01.899161 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:01.899241 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":262,"threads_started":1}
I20260924 01:07:01.983820 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:01.983096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:53629)
0924 01:07:01.983558 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:01.983650 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":239,"threads_started":1}
I20260924 01:07:02.439966 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:02.439207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:42435)
0924 01:07:02.439677 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:02.439775 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
I20260924 01:07:02.901412 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:02.900550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:46849)
0924 01:07:02.901140 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:02.901235 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":327,"threads_started":1}
I20260924 01:07:02.985843 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:02.984936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:42267)
0924 01:07:02.985523 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:02.985622 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":270,"threads_started":1}
I20260924 01:07:03.442030 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:03.440968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:52105)
0924 01:07:03.441634 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:03.441741 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":248,"threads_started":1}
I20260924 01:07:03.903431 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:03.902507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:54929)
0924 01:07:03.903049 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:03.903203 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":245,"threads_started":1}
I20260924 01:07:03.987623 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:03.986924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:41213)
0924 01:07:03.987363 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:03.987457 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260924 01:07:04.443894 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:04.443143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:45547)
0924 01:07:04.443616 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:04.443706 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260924 01:07:04.905344 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:04.904474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:38839)
0924 01:07:04.905019 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:04.905164 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":325,"threads_started":1}
I20260924 01:07:04.989537 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:04.988698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:56127)
0924 01:07:04.989239 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:04.989350 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":256,"threads_started":1}
I20260924 01:07:05.445657 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:05.444926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:56803)
0924 01:07:05.445399 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:05.445482 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
I20260924 01:07:05.907102 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:05.906303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:38653)
0924 01:07:05.906842 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:05.906929 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":232,"threads_started":1}
I20260924 01:07:05.991822 26397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:05.990992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:51859)
0924 01:07:05.991535 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:05.991646 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":279,"threads_started":1}
I20260924 01:07:06.447536 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:06.446782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:43307)
0924 01:07:06.447261 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:06.447374 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260924 01:07:06.908854 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:06.908090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:56387)
0924 01:07:06.908602 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:06.908691 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":260,"threads_started":1}
I20260924 01:07:06.993844 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:06.992916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:53355)
0924 01:07:06.993536 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:06.993642 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":337,"threads_started":1}
I20260924 01:07:07.449424 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:07.448651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:46603)
0924 01:07:07.449152 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:07.449244 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
I20260924 01:07:07.910907 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:07.910011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:47491)
0924 01:07:07.910584 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:07.910715 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":331,"threads_started":1}
I20260924 01:07:07.995484 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:07.994772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:41199)
0924 01:07:07.995235 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:07.995319 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
I20260924 01:07:08.451274 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:08.450507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:57913)
0924 01:07:08.450989 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:08.451080 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260924 01:07:08.912596 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:08.911841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:57709)
0924 01:07:08.912351 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:08.912434 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260924 01:07:08.997331 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:08.996541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:60827)
0924 01:07:08.997026 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:08.997135 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":242,"threads_started":1}
I20260924 01:07:09.453071 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:09.452281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:57873)
0924 01:07:09.452764 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:09.452897 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":234,"threads_started":1}
I20260924 01:07:09.914491 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:09.913598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:53181)
0924 01:07:09.914129 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:09.914300 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":250,"threads_started":1}
I20260924 01:07:09.999130 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:09.998368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:46565)
0924 01:07:09.998830 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:09.998926 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":221,"threads_started":1}
I20260924 01:07:10.454945 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:10.454113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:58369)
0924 01:07:10.454650 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:10.454756 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
I20260924 01:07:10.916355 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:10.915558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:54357)
0924 01:07:10.916061 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:10.916159 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260924 01:07:11.001189 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:11.000407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.65:43909)
0924 01:07:11.000909 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:11.001004 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":234,"threads_started":1}
I20260924 01:07:11.457015 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:11.456230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.66:55743)
0924 01:07:11.456730 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:11.456833 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260924 01:07:11.895206 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 25934
I20260924 01:07:11.918598 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:11.917712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39029 (local address 127.8.254.67:56987)
0924 01:07:11.918246 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:11.918357 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.126:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260924 01:07:11.922767 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:11.896588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.8.254.65:34787)
0924 01:07:11.897298 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:11.897324 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:11.897580 (+ 256us) server_negotiation.cc:407] Connection header received
0924 01:07:11.899587 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:11.899599 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:11.900072 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:11.900380 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:11.901961 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:11.903353 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:11.904609 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:11.905287 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:11.905430 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:11.915126 (+ 9696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:11.915135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:11.915153 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:11.915204 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:11.919086 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:11.919917 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:11.919924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:11.919932 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:11.920100 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:11.920463 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:11.920466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:11.920468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:11.921236 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:11.921473 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:11.921995 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 01:07:11.922472 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":306,"threads_started":1}
I20260924 01:07:11.924206 26099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:12.013500 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 25934
I20260924 01:07:12.031798 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26107
I20260924 01:07:12.049768 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:12.032896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.8.254.66:35423)
0924 01:07:12.033480 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:12.033494 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:12.033533 (+ 39us) server_negotiation.cc:407] Connection header received
0924 01:07:12.035248 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:12.035258 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:12.035705 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:12.036013 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:12.037400 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:12.038761 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:12.040011 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:12.040683 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:12.040816 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:12.043630 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:12.043638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:12.043655 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:12.043706 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:12.046616 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:12.047357 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:12.047365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:12.047372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:12.047541 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:12.047884 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:12.047888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:12.047892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:12.048422 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:12.048655 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:12.049146 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 01:07:12.049483 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260924 01:07:12.051394 26230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:12.136718 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26107
I20260924 01:07:12.155251 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26238
I20260924 01:07:12.174499 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:12.156308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.8.254.67:42615)
0924 01:07:12.156977 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:12.156991 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:12.157034 (+ 43us) server_negotiation.cc:407] Connection header received
0924 01:07:12.158775 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:12.158785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:12.159227 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:12.159530 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:12.161000 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:12.162309 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:12.163622 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:12.164401 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:12.164557 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:12.167496 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:12.167504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:12.167522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:12.167565 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:12.171057 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:12.171796 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:12.171804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:12.171811 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:12.171986 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:12.172338 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:12.172341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:12.172343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:12.172902 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:12.173131 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:12.173522 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 01:07:12.174020 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":326,"threads_started":1}
I20260924 01:07:12.175969 26361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:12.262584 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26238
2026-09-24T01:07:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64485 ms)
[----------] 30 tests from SecurityITest (310011 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:12 dist-test-slave-5c7l krb5kdc[26424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:12 dist-test-slave-5c7l krb5kdc[26424](info): set up 2 sockets
Sep 24 01:07:12 dist-test-slave-5c7l krb5kdc[26424](info): commencing operation
krb5kdc: starting...
W20260924 01:07:13.371659 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:13 dist-test-slave-5c7l krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:07:13.642964 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:46781
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:13.986644 26437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:13.986943 26437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:13.987095 26437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:13.995224 26437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:13.995337 26437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:13.995440 26437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:13.995546 26437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:13.995615 26437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:14.005793 26437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:46781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:14.007261 26437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:14.008595 26437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:14.020565 26443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:14.020648 26442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:14.020658 26445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:14.021673 26437 server_base.cc:1034] running on GCE node
I20260924 01:07:14.022549 26437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:14.023989 26437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:14.025223 26437 hybrid_clock.cc:648] HybridClock initialized: now 1790212034025162 us; error 28 us; skew 500 ppm
I20260924 01:07:14.025542 26437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:14 dist-test-slave-5c7l krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212034, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:14.028846 26437 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:07:14.030491 26437 webserver.cc:460] Webserver started at http://127.8.254.126:33617/ using document root <none> and password file <none>
I20260924 01:07:14.031216 26437 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:14.031347 26437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:14.031689 26437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:14.034789 26437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "f9384d76efc64a77997a85c53cf67093"
format_stamp: "Formatted at 2026-09-24 01:07:14 on dist-test-slave-5c7l"
server_key: "33cb815dcc4d5cd644fbf7051e5990f0"
server_key_iv: "a7421fd03134717591ebe3db99a91a33"
server_key_version: "encryptionkey@0"
I20260924 01:07:14.035552 26437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "f9384d76efc64a77997a85c53cf67093"
format_stamp: "Formatted at 2026-09-24 01:07:14 on dist-test-slave-5c7l"
server_key: "33cb815dcc4d5cd644fbf7051e5990f0"
server_key_iv: "a7421fd03134717591ebe3db99a91a33"
server_key_version: "encryptionkey@0"
I20260924 01:07:14.041155 26437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:07:14.044783 26452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:14.046123 26437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:14.046372 26437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "f9384d76efc64a77997a85c53cf67093"
format_stamp: "Formatted at 2026-09-24 01:07:14 on dist-test-slave-5c7l"
server_key: "33cb815dcc4d5cd644fbf7051e5990f0"
server_key_iv: "a7421fd03134717591ebe3db99a91a33"
server_key_version: "encryptionkey@0"
I20260924 01:07:14.046595 26437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:14.063064 26437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:14.067528 26437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:14.067889 26437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:14.087102 26437 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:46781
I20260924 01:07:14.087136 26503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:46781 every 8 connection(s)
I20260924 01:07:14.088568 26437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:14.092783 26504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:14.098385 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26437
I20260924 01:07:14.098616 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:14.099354 9209 external_mini_cluster.cc:1428] Setting key 19e1ab77e66776fc6ed1dd2f3473bada
I20260924 01:07:14.104811 26504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093: Bootstrap starting.
I20260924 01:07:14.108744 26504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:14.109990 26504 log.cc:824] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:14.113433 26504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093: No bootstrap required, opened a new log
Sep 24 01:07:14 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:14.123554 26504 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9384d76efc64a77997a85c53cf67093" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46781 } }
I20260924 01:07:14.124133 26504 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:14.124262 26504 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9384d76efc64a77997a85c53cf67093, State: Initialized, Role: FOLLOWER
I20260924 01:07:14.124994 26504 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9384d76efc64a77997a85c53cf67093" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46781 } }
I20260924 01:07:14.125289 26504 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:14.125440 26504 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:14.125665 26504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:14.126084 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:14.101533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.8.254.126:46781)
0924 01:07:14.102439 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:14.102455 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:14.102504 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:07:14.104954 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:14.104986 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:14.105510 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:14.105952 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:14.107527 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:14.109554 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:14.111397 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:14.112443 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:14.112646 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:14.116866 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:14.116878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:14.116899 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:14.116956 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:14.121181 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:14.121906 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:14.121915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:14.121925 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:14.122157 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:14.122552 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:14.122557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:14.122561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:14.123281 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:14.123570 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:14.123974 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 01:07:14.124659 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":376,"threads_started":1}
I20260924 01:07:14.128667 26504 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9384d76efc64a77997a85c53cf67093" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46781 } }
I20260924 01:07:14.129251 26504 leader_election.cc:302] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9384d76efc64a77997a85c53cf67093; no voters:
I20260924 01:07:14.129861 26504 leader_election.cc:288] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:14.130000 26509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:14.130798 26509 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [term 1 LEADER]: Becoming Leader. State: Replica: f9384d76efc64a77997a85c53cf67093, State: Running, Role: LEADER
I20260924 01:07:14.131359 26509 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9384d76efc64a77997a85c53cf67093" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46781 } }
I20260924 01:07:14.132055 26504 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:14.134167 26511 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9384d76efc64a77997a85c53cf67093. Latest consensus state: current_term: 1 leader_uuid: "f9384d76efc64a77997a85c53cf67093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9384d76efc64a77997a85c53cf67093" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46781 } } }
I20260924 01:07:14.134415 26510 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9384d76efc64a77997a85c53cf67093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9384d76efc64a77997a85c53cf67093" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46781 } } }
I20260924 01:07:14.134676 26510 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:14.134654 26511 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:14.135653 26515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:14.141479 26515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:14.151353 26515 catalog_manager.cc:1347] Generated new cluster ID: 47090420b9534e92ab601f8c8ab5e931
I20260924 01:07:14.151479 26515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:14.166724 26515 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:14.168109 26515 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:14.184270 26515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093: Generated new TSK 0
I20260924 01:07:14.185245 26515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:07:14.263170 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46781
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:14.606040 26530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:14.606403 26530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:14.606586 26530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:14.613924 26530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:14.614086 26530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:14.614318 26530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:07:14.624186 26530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:14.625520 26530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:14.626979 26530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:14.640651 26535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:14.640580 26538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:14.640694 26536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:14.641834 26530 server_base.cc:1034] running on GCE node
I20260924 01:07:14.642545 26530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:14.643584 26530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:14.644825 26530 hybrid_clock.cc:648] HybridClock initialized: now 1790212034644776 us; error 37 us; skew 500 ppm
I20260924 01:07:14.645171 26530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:14 dist-test-slave-5c7l krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212034, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:14.648537 26530 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:07:14.650107 26530 webserver.cc:460] Webserver started at http://127.8.254.65:37991/ using document root <none> and password file <none>
I20260924 01:07:14.650918 26530 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:14.651053 26530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:14.651391 26530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:14.654558 26530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "67cc851708da4e55875c1d95866758d0"
format_stamp: "Formatted at 2026-09-24 01:07:14 on dist-test-slave-5c7l"
server_key: "42d353bbb8ac26cf10d7fa712d508d92"
server_key_iv: "74e94441bcdf054ca1577041151ac99c"
server_key_version: "encryptionkey@0"
I20260924 01:07:14.655390 26530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "67cc851708da4e55875c1d95866758d0"
format_stamp: "Formatted at 2026-09-24 01:07:14 on dist-test-slave-5c7l"
server_key: "42d353bbb8ac26cf10d7fa712d508d92"
server_key_iv: "74e94441bcdf054ca1577041151ac99c"
server_key_version: "encryptionkey@0"
I20260924 01:07:14.660820 26530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 01:07:14.664511 26545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:14.665939 26530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:14.666126 26530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "67cc851708da4e55875c1d95866758d0"
format_stamp: "Formatted at 2026-09-24 01:07:14 on dist-test-slave-5c7l"
server_key: "42d353bbb8ac26cf10d7fa712d508d92"
server_key_iv: "74e94441bcdf054ca1577041151ac99c"
server_key_version: "encryptionkey@0"
I20260924 01:07:14.666404 26530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:14.694463 26530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:14.698614 26530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:14.698915 26530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:14.699936 26530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:14.701402 26530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:14.701480 26530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:14.701646 26530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:14.701710 26530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:14.735509 26530 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:37099
I20260924 01:07:14.735584 26657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:37099 every 8 connection(s)
I20260924 01:07:14.736835 26530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:07:14.742185 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26530
I20260924 01:07:14.742468 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:07:14.743176 9209 external_mini_cluster.cc:1428] Setting key 68f9799192860ce53afdd05b077aa7b8
Sep 24 01:07:14 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212034, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:14.759393 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:14.739599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:33431 (local address 127.8.254.126:46781)
0924 01:07:14.740085 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:14.740090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:14.740916 (+ 826us) server_negotiation.cc:407] Connection header received
0924 01:07:14.743050 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:14.743056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:14.743154 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:14.743271 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:14.745018 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:14.746083 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:14.747706 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:14.748474 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:14.748622 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:14.752615 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:14.752624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:14.752631 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:14.752680 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:14.755608 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:14.756620 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:14.756624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:14.756627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:14.756791 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:14.757384 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:14.757388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:14.757391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:14.757773 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:14.757914 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:14.758937 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0924 01:07:14.759124 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":270,"threads_started":1}
I20260924 01:07:14.760967 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:14.740060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:46781 (local address 127.8.254.65:33431)
0924 01:07:14.740749 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:14.740793 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:14.742768 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:14.743526 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:14.743534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:14.744231 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:14.744820 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:14.744837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:14.746290 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:14.746294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:14.747551 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:14.747568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:14.748685 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:14.748690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:14.748896 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:14.749618 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:14.749641 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:14.752362 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:14.755770 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:14.755783 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:14.755806 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:14.756494 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:14.756912 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:14.756919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:14.756925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:14.757205 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:14.757943 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:14.757955 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:14.758492 (+ 537us) client_negotiation.cc:769] Sending connection context
0924 01:07:14.758802 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 01:07:14.759411 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":282,"threads_started":1}
I20260924 01:07:14.762704 26658 heartbeater.cc:347] Connected to a master server at 127.8.254.126:46781
I20260924 01:07:14.763048 26658 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:14.763953 26658 heartbeater.cc:511] Master 127.8.254.126:46781 requested a full tablet report, sending...
I20260924 01:07:14.766454 26469 ts_manager.cc:194] Registered new tserver with Master: 67cc851708da4e55875c1d95866758d0 (127.8.254.65:37099)
I20260924 01:07:14.768823 26469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:33431
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:07:14.805508 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46781
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:15.134850 26664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:15.135131 26664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:15.135308 26664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:15.142632 26664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:15.142776 26664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:15.142956 26664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:07:15.153039 26664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:15.154533 26664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:15.155882 26664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:15.169100 26669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:15.169142 26670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:15.169100 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:15.170135 26664 server_base.cc:1034] running on GCE node
I20260924 01:07:15.170894 26664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:15.171938 26664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:15.173164 26664 hybrid_clock.cc:648] HybridClock initialized: now 1790212035173097 us; error 63 us; skew 500 ppm
I20260924 01:07:15.173487 26664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:15 dist-test-slave-5c7l krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212035, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:15.176730 26664 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:07:15.178368 26664 webserver.cc:460] Webserver started at http://127.8.254.66:42679/ using document root <none> and password file <none>
I20260924 01:07:15.179054 26664 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:15.179152 26664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:15.179423 26664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:15.182525 26664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "652298d6382d47c68b9d1d02f62663e2"
format_stamp: "Formatted at 2026-09-24 01:07:15 on dist-test-slave-5c7l"
server_key: "66fa13bdeb86158ea1c9cd489fb671c5"
server_key_iv: "4936107ed5f45b7abab4352d2a3ff8f4"
server_key_version: "encryptionkey@0"
I20260924 01:07:15.183400 26664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "652298d6382d47c68b9d1d02f62663e2"
format_stamp: "Formatted at 2026-09-24 01:07:15 on dist-test-slave-5c7l"
server_key: "66fa13bdeb86158ea1c9cd489fb671c5"
server_key_iv: "4936107ed5f45b7abab4352d2a3ff8f4"
server_key_version: "encryptionkey@0"
I20260924 01:07:15.188869 26664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:07:15.192507 26679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:15.194005 26664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:15.194185 26664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "652298d6382d47c68b9d1d02f62663e2"
format_stamp: "Formatted at 2026-09-24 01:07:15 on dist-test-slave-5c7l"
server_key: "66fa13bdeb86158ea1c9cd489fb671c5"
server_key_iv: "4936107ed5f45b7abab4352d2a3ff8f4"
server_key_version: "encryptionkey@0"
I20260924 01:07:15.194448 26664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:15.270253 26664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:15.274803 26664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:15.275104 26664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:15.276118 26664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:15.277460 26664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:15.277534 26664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:15.277709 26664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:15.277760 26664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:15.312142 26664 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:33755
I20260924 01:07:15.312256 26791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:33755 every 8 connection(s)
I20260924 01:07:15.313508 26664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:07:15.320827 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26664
I20260924 01:07:15.321048 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:07:15.321518 9209 external_mini_cluster.cc:1428] Setting key 4cd03997c1ac3fa48be3e762b59c5bef
Sep 24 01:07:15 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212035, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:15.334057 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.316634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:57797 (local address 127.8.254.126:46781)
0924 01:07:15.317094 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:15.317098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:15.318017 (+ 919us) server_negotiation.cc:407] Connection header received
0924 01:07:15.319900 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:15.319905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:15.319997 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:15.320115 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:15.321647 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.322555 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.324121 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.324547 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.324684 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.328114 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:15.328119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:15.328122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:15.328184 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:15.330877 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:15.331612 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:15.331619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:15.331622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:15.331797 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:15.332227 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:15.332234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:15.332237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:15.332502 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:15.332661 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:15.333425 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 01:07:15.333670 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260924 01:07:15.335065 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.317160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:46781 (local address 127.8.254.66:57797)
0924 01:07:15.317856 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:15.317898 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:15.319638 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:15.320351 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:15.320359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:15.320842 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:15.321480 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.321497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.322713 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.322719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.323963 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.323975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.324772 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.324779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.325008 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.325695 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:15.325718 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:15.327879 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:15.331017 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:15.331027 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:15.331044 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:15.331489 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:15.331914 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:15.331919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:15.331922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:15.332109 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:15.332639 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:15.332647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:15.333031 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 01:07:15.333322 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 01:07:15.333886 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":275,"threads_started":1}
I20260924 01:07:15.336549 26792 heartbeater.cc:347] Connected to a master server at 127.8.254.126:46781
I20260924 01:07:15.336949 26792 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:15.337985 26792 heartbeater.cc:511] Master 127.8.254.126:46781 requested a full tablet report, sending...
I20260924 01:07:15.339811 26469 ts_manager.cc:194] Registered new tserver with Master: 652298d6382d47c68b9d1d02f62663e2 (127.8.254.66:33755)
I20260924 01:07:15.341092 26469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:57797
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:07:15.377975 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46781
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:15.714092 26798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:15.714445 26798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:15.714547 26798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:15.721846 26798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:15.721954 26798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:15.722071 26798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:07:15.731940 26798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:15.733279 26798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:15.734697 26798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:15.749971 26803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:15.749969 26806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:15.750017 26804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:15.751145 26798 server_base.cc:1034] running on GCE node
I20260924 01:07:15.751778 26798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:15.752722 26798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:15.753927 26798 hybrid_clock.cc:648] HybridClock initialized: now 1790212035753879 us; error 39 us; skew 500 ppm
I20260924 01:07:15.754311 26798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:15 dist-test-slave-5c7l krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212035, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:15.757606 26798 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:07:15.759301 26798 webserver.cc:460] Webserver started at http://127.8.254.67:38753/ using document root <none> and password file <none>
I20260924 01:07:15.759967 26798 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:15.760067 26798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:15.760331 26798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:15.763360 26798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "7bfdda444e1743449f9237a2dfed029e"
format_stamp: "Formatted at 2026-09-24 01:07:15 on dist-test-slave-5c7l"
server_key: "66e0697c10c63b1c8fcca5e617235417"
server_key_iv: "798eac98ade6af2b141465957de6d72d"
server_key_version: "encryptionkey@0"
I20260924 01:07:15.764194 26798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "7bfdda444e1743449f9237a2dfed029e"
format_stamp: "Formatted at 2026-09-24 01:07:15 on dist-test-slave-5c7l"
server_key: "66e0697c10c63b1c8fcca5e617235417"
server_key_iv: "798eac98ade6af2b141465957de6d72d"
server_key_version: "encryptionkey@0"
I20260924 01:07:15.769418 26798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:15.772951 26658 heartbeater.cc:503] Master 127.8.254.126:46781 was elected leader, sending a full tablet report...
I20260924 01:07:15.773100 26813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:15.774883 26798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:15.775071 26798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "7bfdda444e1743449f9237a2dfed029e"
format_stamp: "Formatted at 2026-09-24 01:07:15 on dist-test-slave-5c7l"
server_key: "66e0697c10c63b1c8fcca5e617235417"
server_key_iv: "798eac98ade6af2b141465957de6d72d"
server_key_version: "encryptionkey@0"
I20260924 01:07:15.775267 26798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:15.796864 26798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:15.801329 26798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:15.801627 26798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:15.802718 26798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:15.804183 26798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:15.804258 26798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:15.804440 26798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:15.804492 26798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:15.837890 26798 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:36117
I20260924 01:07:15.837954 26925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:36117 every 8 connection(s)
I20260924 01:07:15.839395 26798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:07:15.842563 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26798
I20260924 01:07:15.842774 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:07:15.843292 9209 external_mini_cluster.cc:1428] Setting key 4cca43563aec1136a5e68fcc3d097e3d
Sep 24 01:07:15 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212035, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:15.861323 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.842580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:60127 (local address 127.8.254.126:46781)
0924 01:07:15.843154 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:15.843158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:15.844406 (+ 1248us) server_negotiation.cc:407] Connection header received
0924 01:07:15.846582 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:15.846587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:15.846694 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:15.846816 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:15.848378 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.849164 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.850990 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.851382 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.851562 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.855072 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:15.855077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:15.855080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:15.855113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:15.858082 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:15.859134 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:15.859139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:15.859142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:15.859311 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:15.859764 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:15.859768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:15.859770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:15.860039 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:15.860158 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:15.860884 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 01:07:15.861049 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":343,"threads_started":1}
I20260924 01:07:15.862303 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.843304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:46781 (local address 127.8.254.67:60127)
0924 01:07:15.844245 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:15.844288 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:15.846301 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:15.847069 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:15.847078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:15.847602 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:15.848201 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.848226 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.849334 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.849342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.850813 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.850826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.851637 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.851643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.851851 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.852502 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:15.852525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:15.854872 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:15.858302 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:15.858310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:15.858323 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:15.858890 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:15.859436 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:15.859441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:15.859444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:15.859650 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:15.860150 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:15.860156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:15.860485 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 01:07:15.860761 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 01:07:15.861149 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":364,"threads_started":1}
I20260924 01:07:15.863628 26926 heartbeater.cc:347] Connected to a master server at 127.8.254.126:46781
I20260924 01:07:15.863966 26926 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:15.864657 26926 heartbeater.cc:511] Master 127.8.254.126:46781 requested a full tablet report, sending...
I20260924 01:07:15.866066 26469 ts_manager.cc:194] Registered new tserver with Master: 7bfdda444e1743449f9237a2dfed029e (127.8.254.67:36117)
I20260924 01:07:15.866927 26469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:60127
I20260924 01:07:15.872094 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:07:15.890064 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.876719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.8.254.126:46781)
0924 01:07:15.876905 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:15.876909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:15.877397 (+ 488us) server_negotiation.cc:407] Connection header received
0924 01:07:15.877538 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:15.877541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:15.877607 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:15.877694 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:15.879440 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.880311 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.881754 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.882270 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.882415 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.884947 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:15.884951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:15.884955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:15.884987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:15.887546 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:15.888243 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:15.888248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:15.888252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:15.888372 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:15.888740 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:15.888744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:15.888746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:15.889088 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:15.889280 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:15.889622 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 01:07:15.889842 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:07:15.894418 26469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:07:15.901589 26469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:07:15.925930 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.914464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43794)
0924 01:07:15.916214 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:15.916234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:15.916346 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:15.917405 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:15.917413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:15.917447 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:15.917946 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.917961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.920130 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.920138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.923102 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.923121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.924831 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.924843 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.925038 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.925091 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 01:07:15.925221 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 01:07:15.925432 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"mutex_wait_us":178,"thread_start_us":1474,"threads_started":1}
I20260924 01:07:15.925987 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.915101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.8.254.67:36117)
0924 01:07:15.916607 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:15.916615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:15.916649 (+ 34us) server_negotiation.cc:407] Connection header received
0924 01:07:15.916783 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:15.916794 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:15.917082 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:15.917311 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:15.918868 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.920319 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.923278 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.924632 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.924782 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.925019 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:15.925485 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 01:07:15.925657 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1360,"threads_started":1}
I20260924 01:07:15.926793 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.914252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:36117 (local address 127.0.0.1:58808)
0924 01:07:15.915131 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:15.915170 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:15.915357 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:15.917654 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:15.917662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:15.917699 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:15.918579 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.918595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.920462 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.920467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.923102 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.923121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.924860 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.924868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.925262 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.925304 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 01:07:15.926236 (+ 932us) client_negotiation.cc:240] Negotiation successful
0924 01:07:15.926390 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":521,"thread_start_us":983,"threads_started":1}
I20260924 01:07:15.926923 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.914618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.8.254.65:37099)
0924 01:07:15.916303 (+ 1685us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:15.916311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:15.916341 (+ 30us) server_negotiation.cc:407] Connection header received
0924 01:07:15.916557 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:15.916567 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:15.916860 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:15.917093 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:15.918386 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.920001 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.923330 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.924276 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.924444 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.924693 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:15.926396 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0924 01:07:15.926575 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1326,"threads_started":1}
I20260924 01:07:15.927207 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.916339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:33755 (local address 127.0.0.1:36788)
0924 01:07:15.918960 (+ 2621us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:15.918979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:15.919101 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:15.919701 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:15.919705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:15.919732 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:15.920043 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.920054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.922098 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.922105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.924202 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:15.924219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.926356 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.926362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:15.926540 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.926575 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:07:15.926663 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 01:07:15.926823 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2503,"mutex_wait_us":55,"thread_start_us":2366,"threads_started":1}
I20260924 01:07:15.927897 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:15.916712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.8.254.66:33755)
0924 01:07:15.918258 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:15.918270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:15.919085 (+ 815us) server_negotiation.cc:407] Connection header received
0924 01:07:15.919267 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:15.919275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:15.919500 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:15.919681 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:15.920236 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.921900 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.924387 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:15.925827 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:15.925979 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:15.926217 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:15.927411 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0924 01:07:15.927583 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1658,"threads_started":1}
I20260924 01:07:15.930738 26593 tablet_service.cc:1467] Processing CreateTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f (DEFAULT_TABLE table=test-table [id=5325b4729255444897d6299435b9e6ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:15.930759 26861 tablet_service.cc:1467] Processing CreateTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f (DEFAULT_TABLE table=test-table [id=5325b4729255444897d6299435b9e6ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:15.931288 26727 tablet_service.cc:1467] Processing CreateTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f (DEFAULT_TABLE table=test-table [id=5325b4729255444897d6299435b9e6ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:15.932236 26593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfa42eb0f534a3bbb40b7822584a52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:15.932394 26861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfa42eb0f534a3bbb40b7822584a52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:15.932545 26727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfa42eb0f534a3bbb40b7822584a52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:15.946561 26945 tablet_bootstrap.cc:492] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Bootstrap starting.
I20260924 01:07:15.948765 26946 tablet_bootstrap.cc:492] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Bootstrap starting.
I20260924 01:07:15.949676 26947 tablet_bootstrap.cc:492] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Bootstrap starting.
I20260924 01:07:15.950624 26945 tablet_bootstrap.cc:654] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:15.952122 26945 log.cc:824] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:15.953644 26946 tablet_bootstrap.cc:654] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:15.955063 26946 log.cc:824] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:15.955404 26945 tablet_bootstrap.cc:492] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: No bootstrap required, opened a new log
I20260924 01:07:15.955736 26945 ts_tablet_manager.cc:1397] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 01:07:15.956408 26947 tablet_bootstrap.cc:654] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:15.957639 26947 log.cc:824] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:15.958585 26946 tablet_bootstrap.cc:492] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: No bootstrap required, opened a new log
I20260924 01:07:15.958930 26946 ts_tablet_manager.cc:1397] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 01:07:15.963899 26945 raft_consensus.cc:348] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:15.964298 26945 raft_consensus.cc:374] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:15.964401 26945 raft_consensus.cc:729] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cc851708da4e55875c1d95866758d0, State: Initialized, Role: FOLLOWER
I20260924 01:07:15.965107 26947 tablet_bootstrap.cc:492] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: No bootstrap required, opened a new log
I20260924 01:07:15.965035 26945 consensus_queue.cc:260] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:15.965399 26947 ts_tablet_manager.cc:1397] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260924 01:07:15.967447 26945 ts_tablet_manager.cc:1428] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260924 01:07:15.971356 26946 raft_consensus.cc:348] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:15.971822 26946 raft_consensus.cc:374] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:15.971936 26946 raft_consensus.cc:729] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652298d6382d47c68b9d1d02f62663e2, State: Initialized, Role: FOLLOWER
I20260924 01:07:15.972611 26946 consensus_queue.cc:260] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:15.973857 26792 heartbeater.cc:503] Master 127.8.254.126:46781 was elected leader, sending a full tablet report...
I20260924 01:07:15.974651 26946 ts_tablet_manager.cc:1428] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260924 01:07:15.974900 26947 raft_consensus.cc:348] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:15.975340 26947 raft_consensus.cc:374] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:15.975446 26947 raft_consensus.cc:729] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfdda444e1743449f9237a2dfed029e, State: Initialized, Role: FOLLOWER
I20260924 01:07:15.976065 26947 consensus_queue.cc:260] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:15.977351 26926 heartbeater.cc:503] Master 127.8.254.126:46781 was elected leader, sending a full tablet report...
I20260924 01:07:15.977815 26947 ts_tablet_manager.cc:1428] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
W20260924 01:07:15.989255 26659 tablet.cc:2369] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:07:16.065742 26793 tablet.cc:2369] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 01:07:16.091059 26927 tablet.cc:2369] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:16.206698 26952 raft_consensus.cc:482] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:07:16.207047 26952 raft_consensus.cc:504] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:16.209650 26952 leader_election.cc:288] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bfdda444e1743449f9237a2dfed029e (127.8.254.67:36117), 67cc851708da4e55875c1d95866758d0 (127.8.254.65:37099)
I20260924 01:07:16.215806 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.209463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:44041 (local address 127.8.254.65:37099)
0924 01:07:16.209701 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:16.209708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:16.210280 (+ 572us) server_negotiation.cc:407] Connection header received
0924 01:07:16.210660 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:16.210665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:16.210765 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:16.210878 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:16.211457 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.212185 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.214048 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.214890 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.215075 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.215150 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:16.215428 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 01:07:16.215551 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 01:07:16.215982 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.209283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:46293 (local address 127.8.254.67:36117)
0924 01:07:16.209515 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:16.209523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:16.210730 (+ 1207us) server_negotiation.cc:407] Connection header received
0924 01:07:16.210934 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:16.210940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:16.211025 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:16.211093 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:16.211814 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.212528 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.214266 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.215028 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.215213 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.215290 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:16.215575 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 01:07:16.215703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 01:07:16.215987 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.209214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.8.254.66:44041)
0924 01:07:16.210134 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.210164 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:16.210480 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:16.210911 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:16.210915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:16.210940 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:16.211316 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:16.211327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.212338 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.212346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:16.213894 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:16.213907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.215098 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.215105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:16.215268 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.215292 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 01:07:16.215401 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 01:07:16.215527 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":41,"thread_start_us":410,"threads_started":1}
I20260924 01:07:16.216454 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.209214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:36117 (local address 127.8.254.66:46293)
0924 01:07:16.210606 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.210625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:16.210757 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:16.211303 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:16.211307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:16.211328 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:16.211659 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:16.211666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.212681 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.212686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:16.214082 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:16.214094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.215248 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.215255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:16.215417 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.215443 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 01:07:16.215548 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 01:07:16.215671 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"mutex_wait_us":504,"thread_start_us":549,"threads_started":1}
I20260924 01:07:16.217221 26613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa42eb0f534a3bbb40b7822584a52f" candidate_uuid: "652298d6382d47c68b9d1d02f62663e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cc851708da4e55875c1d95866758d0" is_pre_election: true
I20260924 01:07:16.217659 26881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa42eb0f534a3bbb40b7822584a52f" candidate_uuid: "652298d6382d47c68b9d1d02f62663e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfdda444e1743449f9237a2dfed029e" is_pre_election: true
I20260924 01:07:16.217878 26613 raft_consensus.cc:2454] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652298d6382d47c68b9d1d02f62663e2 in term 0.
I20260924 01:07:16.218251 26881 raft_consensus.cc:2454] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652298d6382d47c68b9d1d02f62663e2 in term 0.
I20260924 01:07:16.218695 26682 leader_election.cc:302] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652298d6382d47c68b9d1d02f62663e2, 67cc851708da4e55875c1d95866758d0; no voters:
I20260924 01:07:16.219153 26952 raft_consensus.cc:2781] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:07:16.219334 26952 raft_consensus.cc:482] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:07:16.219460 26952 raft_consensus.cc:3037] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:16.221594 26952 raft_consensus.cc:504] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:16.222250 26952 leader_election.cc:288] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7bfdda444e1743449f9237a2dfed029e (127.8.254.67:36117), 67cc851708da4e55875c1d95866758d0 (127.8.254.65:37099)
I20260924 01:07:16.222657 26881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa42eb0f534a3bbb40b7822584a52f" candidate_uuid: "652298d6382d47c68b9d1d02f62663e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfdda444e1743449f9237a2dfed029e"
I20260924 01:07:16.222783 26613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa42eb0f534a3bbb40b7822584a52f" candidate_uuid: "652298d6382d47c68b9d1d02f62663e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cc851708da4e55875c1d95866758d0"
I20260924 01:07:16.222895 26881 raft_consensus.cc:3037] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:16.222996 26613 raft_consensus.cc:3037] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:16.226241 26881 raft_consensus.cc:2454] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652298d6382d47c68b9d1d02f62663e2 in term 1.
I20260924 01:07:16.226239 26613 raft_consensus.cc:2454] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652298d6382d47c68b9d1d02f62663e2 in term 1.
I20260924 01:07:16.226744 26680 leader_election.cc:302] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 652298d6382d47c68b9d1d02f62663e2, 67cc851708da4e55875c1d95866758d0, 7bfdda444e1743449f9237a2dfed029e; no voters:
I20260924 01:07:16.227159 26952 raft_consensus.cc:2781] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:16.227624 26952 raft_consensus.cc:686] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 1 LEADER]: Becoming Leader. State: Replica: 652298d6382d47c68b9d1d02f62663e2, State: Running, Role: LEADER
I20260924 01:07:16.228329 26952 consensus_queue.cc:237] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } }
I20260924 01:07:16.232895 26469 catalog_manager.cc:5495] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652298d6382d47c68b9d1d02f62663e2 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "652298d6382d47c68b9d1d02f62663e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652298d6382d47c68b9d1d02f62663e2" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 33755 } health_report { overall_health: HEALTHY } } }
I20260924 01:07:16.294310 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.288959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.8.254.66:33755)
0924 01:07:16.289139 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:16.289144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:16.289203 (+ 59us) server_negotiation.cc:407] Connection header received
0924 01:07:16.289399 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:16.289403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:16.289484 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:16.289607 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:07:16.290015 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.290782 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.292275 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.292673 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.292761 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.293206 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:07:16.293728 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:07:16.293840 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:16.293980 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 01:07:16.294047 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:07:16.303017 26613 raft_consensus.cc:1261] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 1 FOLLOWER]: Refusing update from remote peer 652298d6382d47c68b9d1d02f62663e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:16.303026 26881 raft_consensus.cc:1261] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 1 FOLLOWER]: Refusing update from remote peer 652298d6382d47c68b9d1d02f62663e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:16.304044 26956 consensus_queue.cc:1035] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfdda444e1743449f9237a2dfed029e" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:16.304437 26952 consensus_queue.cc:1035] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67cc851708da4e55875c1d95866758d0" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:16.316763 26959 mvcc.cc:205] Tried to move back new op lower bound from 7332708500686884864 to 7332708500395016192. Current Snapshot: MvccSnapshot[applied={T|T < 7332708500686884864}]
I20260924 01:07:16.317863 26961 mvcc.cc:205] Tried to move back new op lower bound from 7332708500686884864 to 7332708500395016192. Current Snapshot: MvccSnapshot[applied={T|T < 7332708500686884864}]
I20260924 01:07:16.333583 26468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:16.334020 26468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:16.338877 26468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9384d76efc64a77997a85c53cf67093: Sending DeleteTablet for 3 replicas of tablet bcfa42eb0f534a3bbb40b7822584a52f
I20260924 01:07:16.340432 26861 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:16 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:58808
I20260924 01:07:16.340674 26592 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:16 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:43794
I20260924 01:07:16.340943 26593 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:16 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:43794
I20260924 01:07:16.341504 26727 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfa42eb0f534a3bbb40b7822584a52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:16 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:36788
I20260924 01:07:16.342516 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26530
I20260924 01:07:16.368909 26971 tablet_replica.cc:330] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: stopping tablet replica
I20260924 01:07:16.369587 26971 raft_consensus.cc:2229] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:07:16.370891 26970 tablet_replica.cc:330] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: stopping tablet replica
I20260924 01:07:16.371482 26970 raft_consensus.cc:2229] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:16.372154 26970 raft_consensus.cc:2258] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 01:07:16 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:07:16.374648 26971 raft_consensus.cc:2258] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:16.378809 26971 ts_tablet_manager.cc:1905] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:16.382710 26969 tablet_replica.cc:330] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: stopping tablet replica
I20260924 01:07:16.383164 26969 raft_consensus.cc:2229] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:16.383656 26969 raft_consensus.cc:2258] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:16.382094 26970 ts_tablet_manager.cc:1905] T bcfa42eb0f534a3bbb40b7822584a52f P 67cc851708da4e55875c1d95866758d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:16.384728 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.343766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.8.254.65:37099)
0924 01:07:16.343982 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:16.343992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:16.362833 (+ 18841us) server_negotiation.cc:407] Connection header received
0924 01:07:16.363199 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:16.363204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:16.363297 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:16.363415 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:16.365315 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.366528 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.368582 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.369331 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.369464 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.375966 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:16.375986 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:16.376005 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:16.376052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:16.380399 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:16.381394 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:16.381406 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:16.381411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:16.381591 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:16.382057 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:16.382062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:16.382065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:16.382676 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:16.383058 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:16.383954 (+ 896us) server_negotiation.cc:299] Negotiation successful
0924 01:07:16.384286 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 01:07:16.386511 26653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:16.390745 26971 ts_tablet_manager.cc:1918] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:16.390931 26971 log.cc:1196] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/bcfa42eb0f534a3bbb40b7822584a52f
I20260924 01:07:16.391520 26971 ts_tablet_manager.cc:1939] T bcfa42eb0f534a3bbb40b7822584a52f P 652298d6382d47c68b9d1d02f62663e2: Deleting consensus metadata
I20260924 01:07:16.391729 26969 ts_tablet_manager.cc:1905] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:16.393465 26455 catalog_manager.cc:4841] TS 652298d6382d47c68b9d1d02f62663e2 (127.8.254.66:33755): tablet bcfa42eb0f534a3bbb40b7822584a52f (table test-table [id=5325b4729255444897d6299435b9e6ee]) successfully deleted
I20260924 01:07:16.398957 26969 ts_tablet_manager.cc:1918] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:16.399137 26969 log.cc:1196] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/bcfa42eb0f534a3bbb40b7822584a52f
I20260924 01:07:16.399708 26969 ts_tablet_manager.cc:1939] T bcfa42eb0f534a3bbb40b7822584a52f P 7bfdda444e1743449f9237a2dfed029e: Deleting consensus metadata
I20260924 01:07:16.401698 26453 catalog_manager.cc:4841] TS 7bfdda444e1743449f9237a2dfed029e (127.8.254.67:36117): tablet bcfa42eb0f534a3bbb40b7822584a52f (table test-table [id=5325b4729255444897d6299435b9e6ee]) successfully deleted
I20260924 01:07:16.488461 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26530
W20260924 01:07:16.505852 26455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.254.65:37099 (error 108)
I20260924 01:07:16.506078 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26664
I20260924 01:07:16.508728 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.507946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43806)
0924 01:07:16.508434 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.508554 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":265,"threads_started":1}
W20260924 01:07:16.509075 26455 catalog_manager.cc:4584] TS 67cc851708da4e55875c1d95866758d0 (127.8.254.65:37099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcfa42eb0f534a3bbb40b7822584a52f: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Sep 24 01:07:16 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:07:16.525141 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.506916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.8.254.66:33755)
0924 01:07:16.507114 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:16.507121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:16.507272 (+ 151us) server_negotiation.cc:407] Connection header received
0924 01:07:16.507426 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:16.507432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:16.507531 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:16.507635 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:16.509425 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.510434 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.512395 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.512885 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.512991 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.518429 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:16.518450 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:16.518466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:16.518518 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:16.521869 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:16.522748 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:16.522766 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:16.522769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:16.522937 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:16.523387 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:16.523392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:16.523395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:16.523781 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:16.524240 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:16.524481 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 01:07:16.524735 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:07:16.526010 26787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:16.553673 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.553221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43818)
0924 01:07:16.553423 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.553515 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 01:07:16.554824 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.554403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43830)
0924 01:07:16.554584 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.554679 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 01:07:16.563329 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.562868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43834)
0924 01:07:16.563072 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.563161 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 01:07:16.564427 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.564079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43842)
0924 01:07:16.564221 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.564291 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 01:07:16.603991 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.603491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43848)
0924 01:07:16.603739 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.603838 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 01:07:16.605168 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.604758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43850)
0924 01:07:16.604958 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.605038 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 01:07:16.641662 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.641143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43866)
0924 01:07:16.641411 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.641511 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 01:07:16.642773 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.642408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43874)
0924 01:07:16.642556 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.642627 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 01:07:16.645646 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26664
I20260924 01:07:16.665162 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26798
Sep 24 01:07:16 dist-test-slave-5c7l krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:07:16.680898 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.665917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.8.254.67:36117)
0924 01:07:16.666092 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:16.666096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:16.666168 (+ 72us) server_negotiation.cc:407] Connection header received
0924 01:07:16.666409 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:16.666412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:16.666482 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:16.666578 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:16.667842 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.668540 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.669878 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:16.670332 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:16.670436 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:16.674949 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:16.674965 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:16.674982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:16.675016 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:16.677690 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:16.678868 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:16.678876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:16.678878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:16.679030 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:16.679476 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:16.679480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:16.679482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:16.679899 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:16.680189 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:16.680421 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 01:07:16.680588 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:07:16.681522 26921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:16.706348 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.705789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43884)
0924 01:07:16.706020 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.706115 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 01:07:16.707419 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.707052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43890)
0924 01:07:16.707207 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.707283 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 01:07:16.745011 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.744473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43898)
0924 01:07:16.744724 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.744852 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 01:07:16.746086 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.745733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43914)
0924 01:07:16.745869 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.745951 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 01:07:16.772408 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26798
I20260924 01:07:16.789448 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26437
I20260924 01:07:16.790665 26499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:16.902580 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26437
I20260924 01:07:16.902614 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:16.902027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37099 (local address 127.0.0.1:43916)
0924 01:07:16.902340 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:16.902443 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.65:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
2026-09-24T01:07:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4647 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T01:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:16Z Disabled control of system clock
I20260924 01:07:16.985410 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:41837
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:41837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 01:07:17.319624 26982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:17.319917 26982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:17.320077 26982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:17.327283 26982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:17.327404 26982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:17.327500 26982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:17.327605 26982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:17.327682 26982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:17.336964 26982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:41837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:17.338387 26982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:17.340164 26982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:17.353354 26987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:17.353390 26988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:17.353317 26990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:17.354252 26982 server_base.cc:1034] running on GCE node
I20260924 01:07:17.355077 26982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:17.356592 26982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:17.357856 26982 hybrid_clock.cc:648] HybridClock initialized: now 1790212037357801 us; error 44 us; skew 500 ppm
I20260924 01:07:17.358299 26982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:17.370600 26982 webserver.cc:460] Webserver started at http://127.8.254.126:40619/ using document root <none> and password file <none>
I20260924 01:07:17.371415 26982 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:17.371570 26982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:17.371938 26982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:17.375259 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a"
format_stamp: "Formatted at 2026-09-24 01:07:17 on dist-test-slave-5c7l"
server_key: "6a07b28728ea9b481cab96e6c14423bd"
server_key_iv: "48ef6a6cfb0f02968afce2ec82c8f214"
server_key_version: "encryptionkey@0"
I20260924 01:07:17.376119 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a"
format_stamp: "Formatted at 2026-09-24 01:07:17 on dist-test-slave-5c7l"
server_key: "6a07b28728ea9b481cab96e6c14423bd"
server_key_iv: "48ef6a6cfb0f02968afce2ec82c8f214"
server_key_version: "encryptionkey@0"
I20260924 01:07:17.381623 26982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:17.385265 26996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:17.386659 26982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:17.386859 26982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a"
format_stamp: "Formatted at 2026-09-24 01:07:17 on dist-test-slave-5c7l"
server_key: "6a07b28728ea9b481cab96e6c14423bd"
server_key_iv: "48ef6a6cfb0f02968afce2ec82c8f214"
server_key_version: "encryptionkey@0"
I20260924 01:07:17.387080 26982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:17.402338 26982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:17.403270 26982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:17.403596 26982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:17.422899 26982 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:41837
I20260924 01:07:17.422937 27047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:41837 every 8 connection(s)
I20260924 01:07:17.424372 26982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:17.428423 27048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:17.429936 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 26982
I20260924 01:07:17.430111 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:17.430663 9209 external_mini_cluster.cc:1428] Setting key 402d98ad02c0b1623681bccceb6e0997
I20260924 01:07:17.440649 27048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a: Bootstrap starting.
I20260924 01:07:17.444460 27048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:17.445235 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:17.432679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.8.254.126:41837)
0924 01:07:17.433802 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:17.433816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:17.433864 (+ 48us) server_negotiation.cc:407] Connection header received
0924 01:07:17.435651 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:17.435674 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:17.436137 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:17.436467 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:17.437123 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:17.438536 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:17.440488 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:17.441217 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:17.441366 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:17.442756 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:17.442764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:17.442778 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:17.443311 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:17.443415 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:17.443437 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:17.443529 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:17.443778 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 01:07:17.444127 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":418,"threads_started":1}
I20260924 01:07:17.445695 27048 log.cc:824] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:17.448812 27048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a: No bootstrap required, opened a new log
I20260924 01:07:17.456133 27048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41837 } }
I20260924 01:07:17.456475 27048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:17.456547 27048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4b3b53d8d34bb4bffe4294f5f4044a, State: Initialized, Role: FOLLOWER
I20260924 01:07:17.457026 27048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41837 } }
I20260924 01:07:17.457226 27048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:17.457319 27048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:17.457439 27048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:17.459563 27048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41837 } }
I20260924 01:07:17.460016 27048 leader_election.cc:302] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4b3b53d8d34bb4bffe4294f5f4044a; no voters:
I20260924 01:07:17.460731 27048 leader_election.cc:288] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:17.460889 27053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:17.461732 27053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [term 1 LEADER]: Becoming Leader. State: Replica: 4b4b3b53d8d34bb4bffe4294f5f4044a, State: Running, Role: LEADER
I20260924 01:07:17.462337 27053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41837 } }
I20260924 01:07:17.463053 27048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:17.465139 27054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41837 } } }
I20260924 01:07:17.465523 27054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:17.465108 27055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4b3b53d8d34bb4bffe4294f5f4044a. Latest consensus state: current_term: 1 leader_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4b3b53d8d34bb4bffe4294f5f4044a" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41837 } } }
I20260924 01:07:17.466966 27055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:17.469200 27058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:17.476006 27058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:17.485211 27058 catalog_manager.cc:1347] Generated new cluster ID: 79f24ff8ed4e4e2996fb6d5fa3f839fb
I20260924 01:07:17.485416 27058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:17.507594 27058 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:17.509038 27058 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:17.516956 27058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a: Generated new TSK 0
I20260924 01:07:17.517820 27058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:17.526552 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41837
--builtin_ntp_servers=127.8.254.84:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 01:07:17.882057 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:17.882397 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:17.882570 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:17.889813 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:17.889910 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:17.890077 27072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:07:17.899447 27072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:17.900866 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:17.902664 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:17.915795 27077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:17.915720 27078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:17.915771 27080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:17.916163 27072 server_base.cc:1034] running on GCE node
I20260924 01:07:17.917016 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:17.917932 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:17.919140 27072 hybrid_clock.cc:648] HybridClock initialized: now 1790212037919079 us; error 49 us; skew 500 ppm
I20260924 01:07:17.919485 27072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:17.921772 27072 webserver.cc:460] Webserver started at http://127.8.254.65:41009/ using document root <none> and password file <none>
I20260924 01:07:17.922588 27072 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:17.922705 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:17.923045 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:17.926039 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "7d9ac384bf5f41df8b55eb5f76979ba4"
format_stamp: "Formatted at 2026-09-24 01:07:17 on dist-test-slave-5c7l"
server_key: "73ae68a87bb165cb9a9d4974cd422a4b"
server_key_iv: "9170b4c8715f0a50f56f4d439df0d967"
server_key_version: "encryptionkey@0"
I20260924 01:07:17.927006 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "7d9ac384bf5f41df8b55eb5f76979ba4"
format_stamp: "Formatted at 2026-09-24 01:07:17 on dist-test-slave-5c7l"
server_key: "73ae68a87bb165cb9a9d4974cd422a4b"
server_key_iv: "9170b4c8715f0a50f56f4d439df0d967"
server_key_version: "encryptionkey@0"
I20260924 01:07:17.933058 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 01:07:17.936754 27086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:17.938153 27072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:17.938359 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "7d9ac384bf5f41df8b55eb5f76979ba4"
format_stamp: "Formatted at 2026-09-24 01:07:17 on dist-test-slave-5c7l"
server_key: "73ae68a87bb165cb9a9d4974cd422a4b"
server_key_iv: "9170b4c8715f0a50f56f4d439df0d967"
server_key_version: "encryptionkey@0"
I20260924 01:07:17.938588 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:17.961690 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:17.962742 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:17.963076 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:17.964226 27072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:17.965809 27072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:17.965924 27072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:17.966058 27072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:17.966166 27072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:17.999526 27072 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:41217
I20260924 01:07:17.999619 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:41217 every 8 connection(s)
I20260924 01:07:18.001005 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:07:18.002854 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27072
I20260924 01:07:18.003031 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:07:18.003458 9209 external_mini_cluster.cc:1428] Setting key 59844282519b4fe1b0b7635ee7680061
I20260924 01:07:18.006321 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41837
--builtin_ntp_servers=127.8.254.84:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 01:07:18.015116 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.003777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:50083 (local address 127.8.254.126:41837)
0924 01:07:18.004285 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.004291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:18.005678 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 01:07:18.007648 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:18.007656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:18.007764 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:18.007888 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:18.008961 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.010103 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.011703 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.012457 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.012605 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.014162 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:18.014167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:18.014170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:18.014228 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:18.014282 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:18.014296 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:18.014403 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:18.014638 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 01:07:18.014754 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260924 01:07:18.016285 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.004440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41837 (local address 127.8.254.65:50083)
0924 01:07:18.005508 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:18.005556 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:18.007372 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:18.008092 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:18.008100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:18.008206 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:18.008776 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.008793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.010307 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.010310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:18.011548 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.011560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.012714 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.012719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:18.012990 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.013798 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:18.013801 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:18.013842 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:18.013939 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:18.014404 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:18.014418 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:18.014429 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 01:07:18.014621 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 01:07:18.015054 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":338,"threads_started":1}
I20260924 01:07:18.017956 27199 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41837
I20260924 01:07:18.018334 27199 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:18.019133 27199 heartbeater.cc:511] Master 127.8.254.126:41837 requested a full tablet report, sending...
I20260924 01:07:18.021469 27013 ts_manager.cc:194] Registered new tserver with Master: 7d9ac384bf5f41df8b55eb5f76979ba4 (127.8.254.65:41217)
I20260924 01:07:18.023672 27013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.65:50083
W20260924 01:07:18.344158 27203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:18.344448 27203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:18.344539 27203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:18.352563 27203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:18.352671 27203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:18.352780 27203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:07:18.361954 27203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:18.363343 27203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:18.365198 27203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:18.378494 27208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:18.378544 27209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:18.378458 27211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:18.384071 27203 server_base.cc:1034] running on GCE node
I20260924 01:07:18.384822 27203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:18.385813 27203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:18.387089 27203 hybrid_clock.cc:648] HybridClock initialized: now 1790212038387039 us; error 38 us; skew 500 ppm
I20260924 01:07:18.387471 27203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:18.389789 27203 webserver.cc:460] Webserver started at http://127.8.254.66:43593/ using document root <none> and password file <none>
I20260924 01:07:18.390698 27203 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:18.390834 27203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:18.391170 27203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:18.394124 27203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "502a8ff222034405ac3006bb36686784"
format_stamp: "Formatted at 2026-09-24 01:07:18 on dist-test-slave-5c7l"
server_key: "fffb5c9c18c17e491d4e8d8cfc61ad27"
server_key_iv: "4f762bed2f20c6dd74de1185e6408e74"
server_key_version: "encryptionkey@0"
I20260924 01:07:18.395069 27203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "502a8ff222034405ac3006bb36686784"
format_stamp: "Formatted at 2026-09-24 01:07:18 on dist-test-slave-5c7l"
server_key: "fffb5c9c18c17e491d4e8d8cfc61ad27"
server_key_iv: "4f762bed2f20c6dd74de1185e6408e74"
server_key_version: "encryptionkey@0"
I20260924 01:07:18.400063 27203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 01:07:18.403704 27217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:18.405040 27203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:07:18.405231 27203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "502a8ff222034405ac3006bb36686784"
format_stamp: "Formatted at 2026-09-24 01:07:18 on dist-test-slave-5c7l"
server_key: "fffb5c9c18c17e491d4e8d8cfc61ad27"
server_key_iv: "4f762bed2f20c6dd74de1185e6408e74"
server_key_version: "encryptionkey@0"
I20260924 01:07:18.405462 27203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:18.432317 27203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:18.433296 27203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:18.433604 27203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:18.434768 27203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:18.436654 27203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:18.436771 27203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:18.436878 27203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:18.436949 27203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:18.473387 27203 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:36131
I20260924 01:07:18.473470 27329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:36131 every 8 connection(s)
I20260924 01:07:18.474843 27203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:07:18.484974 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27203
I20260924 01:07:18.485173 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:07:18.485664 9209 external_mini_cluster.cc:1428] Setting key d5d176b632eb54633764a7a6d64b870d
I20260924 01:07:18.487442 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.477426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:38339 (local address 127.8.254.126:41837)
0924 01:07:18.477607 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.477611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:18.478962 (+ 1351us) server_negotiation.cc:407] Connection header received
0924 01:07:18.480755 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:18.480761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:18.480855 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:18.480967 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:18.482015 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.482826 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.484377 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.484958 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.485099 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.486586 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:18.486592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:18.486596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:18.486616 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:18.486655 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:18.486672 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:18.486771 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:18.487013 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 01:07:18.487110 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:07:18.488677 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.477863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41837 (local address 127.8.254.66:38339)
0924 01:07:18.478805 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:18.478840 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:18.480484 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:18.481197 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:18.481205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:18.481299 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:18.481837 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.481863 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.482981 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.482994 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:18.484220 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.484232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.485280 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.485290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:18.485498 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.486268 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:18.486276 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:18.486316 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:18.486393 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:18.486791 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:18.486802 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:18.486817 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 01:07:18.486998 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 01:07:18.487382 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":311,"threads_started":1}
I20260924 01:07:18.488858 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41837
--builtin_ntp_servers=127.8.254.84:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 01:07:18.490164 27330 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41837
I20260924 01:07:18.490550 27330 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:18.491431 27330 heartbeater.cc:511] Master 127.8.254.126:41837 requested a full tablet report, sending...
I20260924 01:07:18.492930 27013 ts_manager.cc:194] Registered new tserver with Master: 502a8ff222034405ac3006bb36686784 (127.8.254.66:36131)
I20260924 01:07:18.494169 27013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.66:38339
W20260924 01:07:18.819178 27333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:18.819523 27333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:18.819695 27333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:18.826946 27333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:18.827098 27333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:18.827292 27333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:07:18.836476 27333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:18.837872 27333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:18.839694 27333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:18.853135 27338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:18.853083 27339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:18.853049 27341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:18.854171 27333 server_base.cc:1034] running on GCE node
I20260924 01:07:18.854815 27333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:18.855770 27333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:18.857031 27333 hybrid_clock.cc:648] HybridClock initialized: now 1790212038856978 us; error 64 us; skew 500 ppm
I20260924 01:07:18.857357 27333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:18.859754 27333 webserver.cc:460] Webserver started at http://127.8.254.67:35135/ using document root <none> and password file <none>
I20260924 01:07:18.860477 27333 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:18.860615 27333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:18.860950 27333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:18.864037 27333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "65658aa7935544b68919aab9c84dbb38"
format_stamp: "Formatted at 2026-09-24 01:07:18 on dist-test-slave-5c7l"
server_key: "72957014e96a8d852f32d03e31b71c84"
server_key_iv: "6570abaa97b7e6459ef83fe5c0bba2f0"
server_key_version: "encryptionkey@0"
I20260924 01:07:18.864926 27333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "65658aa7935544b68919aab9c84dbb38"
format_stamp: "Formatted at 2026-09-24 01:07:18 on dist-test-slave-5c7l"
server_key: "72957014e96a8d852f32d03e31b71c84"
server_key_iv: "6570abaa97b7e6459ef83fe5c0bba2f0"
server_key_version: "encryptionkey@0"
I20260924 01:07:18.870078 27333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 01:07:18.873663 27347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:18.875030 27333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:18.875217 27333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "65658aa7935544b68919aab9c84dbb38"
format_stamp: "Formatted at 2026-09-24 01:07:18 on dist-test-slave-5c7l"
server_key: "72957014e96a8d852f32d03e31b71c84"
server_key_iv: "6570abaa97b7e6459ef83fe5c0bba2f0"
server_key_version: "encryptionkey@0"
I20260924 01:07:18.875452 27333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:18.894870 27333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:18.895674 27333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:18.896073 27333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:18.897038 27333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:18.898583 27333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:18.898716 27333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:18.898859 27333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:18.898962 27333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:18.932165 27333 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:40163
I20260924 01:07:18.932221 27459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:40163 every 8 connection(s)
I20260924 01:07:18.933563 27333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:07:18.934051 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27333
I20260924 01:07:18.934283 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:07:18.934969 9209 external_mini_cluster.cc:1428] Setting key 58bf5a3ec340a7af0518fa141b9d36ae
I20260924 01:07:18.947046 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.936334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:37757 (local address 127.8.254.126:41837)
0924 01:07:18.936525 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.936531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:18.938234 (+ 1703us) server_negotiation.cc:407] Connection header received
0924 01:07:18.940110 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:18.940117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:18.940203 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:18.940322 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:18.941420 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.942421 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.944201 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.944650 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.944801 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.946140 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:18.946147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:18.946158 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:18.946180 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:18.946256 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:18.946272 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:18.946384 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:18.946608 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 01:07:18.946712 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 01:07:18.948098 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.936939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41837 (local address 127.8.254.67:37757)
0924 01:07:18.938034 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:18.938099 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:18.939826 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:18.940565 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:18.940575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:18.940664 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:18.941246 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.941263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.942581 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.942587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:18.944040 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.944054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.944896 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.944903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:18.945139 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.945791 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:18.945793 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:18.945833 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:18.945935 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:18.946382 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:18.946395 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:18.946405 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 01:07:18.946596 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 01:07:18.947025 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":289,"threads_started":1}
I20260924 01:07:18.949712 27460 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41837
I20260924 01:07:18.950083 27460 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:18.950997 27460 heartbeater.cc:511] Master 127.8.254.126:41837 requested a full tablet report, sending...
I20260924 01:07:18.952311 27012 ts_manager.cc:194] Registered new tserver with Master: 65658aa7935544b68919aab9c84dbb38 (127.8.254.67:40163)
I20260924 01:07:18.953112 27012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.67:37757
I20260924 01:07:18.963428 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:07:18.973147 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.967540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.8.254.126:41837)
0924 01:07:18.967741 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.967746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:18.968113 (+ 367us) server_negotiation.cc:407] Connection header received
0924 01:07:18.968250 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:18.968254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:18.968327 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:18.968426 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:18.968884 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.969570 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.971014 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:18.971454 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:18.971587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:18.972560 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:18.972565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:18.972568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:18.972586 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:18.972618 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:18.972630 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:18.972699 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:18.972821 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 01:07:18.972897 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:07:18.977070 27012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:07:18.983986 27012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:07:19.007653 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.995824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:41217 (local address 127.0.0.1:57748)
0924 01:07:18.996822 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:18.996895 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:18.997110 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:18.998479 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:18.998487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:18.998523 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:18.998975 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:18.998989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.001249 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.001255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.004263 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.004279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.006704 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.006713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.006952 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.006993 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 01:07:19.007082 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 01:07:19.007277 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":253,"thread_start_us":453,"threads_started":1}
I20260924 01:07:19.007746 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.996663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.8.254.66:36131)
0924 01:07:18.998150 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.998159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:18.998984 (+ 825us) server_negotiation.cc:407] Connection header received
0924 01:07:18.999089 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:18.999096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:18.999330 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:18.999518 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:19.000497 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.001989 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.004637 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.006099 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.006262 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.006502 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.007211 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.007402 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":1200,"threads_started":1}
I20260924 01:07:19.008558 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.996077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.8.254.65:41217)
0924 01:07:18.997366 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.997379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:18.997417 (+ 38us) server_negotiation.cc:407] Connection header received
0924 01:07:18.997532 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:18.997544 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:18.997865 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:18.998075 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:18.999316 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.001069 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.004521 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.006035 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.006184 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.006506 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.008009 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.008173 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":955,"threads_started":1}
I20260924 01:07:19.008795 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.997927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:40163 (local address 127.0.0.1:57502)
0924 01:07:19.000081 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:19.000105 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:19.000228 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:19.001084 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:19.001089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:19.001116 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:19.001542 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.001554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.003535 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.003541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.005713 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.005731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.008028 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.008034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.008195 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.008227 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:07:19.008316 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 01:07:19.008439 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2019,"mutex_wait_us":5,"thread_start_us":5368,"threads_started":1}
I20260924 01:07:19.009302 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.996059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:36131 (local address 127.0.0.1:55160)
0924 01:07:18.998675 (+ 2616us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:18.998703 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:18.998840 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:18.999769 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:18.999777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:18.999805 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:19.000289 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.000306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.002158 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.002163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.004426 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.004442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.006654 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.006663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.006864 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.006940 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 01:07:19.007060 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 01:07:19.008793 (+ 1733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2432,"mutex_wait_us":1578,"thread_start_us":2252,"threads_started":1}
I20260924 01:07:19.010093 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:18.998522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.8.254.67:40163)
0924 01:07:18.999655 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:18.999664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.000350 (+ 686us) server_negotiation.cc:407] Connection header received
0924 01:07:19.000447 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.000454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.000659 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.000837 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:19.001769 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.003366 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.005957 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.007444 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.007568 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.009485 (+ 1917us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.009569 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.009747 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":4819,"threads_started":1}
I20260924 01:07:19.012980 27395 tablet_service.cc:1467] Processing CreateTablet for tablet 8f09232fab6a41e590a193d6bf7234ea (DEFAULT_TABLE table=test-table [id=db6b1ff26ff64f10ad2cd12509c176dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:19.012984 27134 tablet_service.cc:1467] Processing CreateTablet for tablet 8f09232fab6a41e590a193d6bf7234ea (DEFAULT_TABLE table=test-table [id=db6b1ff26ff64f10ad2cd12509c176dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:19.012984 27265 tablet_service.cc:1467] Processing CreateTablet for tablet 8f09232fab6a41e590a193d6bf7234ea (DEFAULT_TABLE table=test-table [id=db6b1ff26ff64f10ad2cd12509c176dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:19.014428 27395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f09232fab6a41e590a193d6bf7234ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:19.014442 27265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f09232fab6a41e590a193d6bf7234ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:19.014433 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f09232fab6a41e590a193d6bf7234ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:19.026824 27478 tablet_bootstrap.cc:492] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Bootstrap starting.
I20260924 01:07:19.026932 27199 heartbeater.cc:503] Master 127.8.254.126:41837 was elected leader, sending a full tablet report...
I20260924 01:07:19.029412 27479 tablet_bootstrap.cc:492] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Bootstrap starting.
I20260924 01:07:19.030767 27478 tablet_bootstrap.cc:654] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:19.030937 27480 tablet_bootstrap.cc:492] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Bootstrap starting.
I20260924 01:07:19.032035 27478 log.cc:824] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:19.034149 27479 tablet_bootstrap.cc:654] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:19.034381 27480 tablet_bootstrap.cc:654] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:19.035306 27478 tablet_bootstrap.cc:492] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: No bootstrap required, opened a new log
I20260924 01:07:19.035493 27480 log.cc:824] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:19.035493 27479 log.cc:824] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:19.035584 27478 ts_tablet_manager.cc:1397] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 01:07:19.038337 27480 tablet_bootstrap.cc:492] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: No bootstrap required, opened a new log
I20260924 01:07:19.038604 27480 ts_tablet_manager.cc:1397] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 01:07:19.038693 27479 tablet_bootstrap.cc:492] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: No bootstrap required, opened a new log
I20260924 01:07:19.038997 27479 ts_tablet_manager.cc:1397] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 01:07:19.046319 27480 raft_consensus.cc:348] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.046314 27478 raft_consensus.cc:348] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.046779 27478 raft_consensus.cc:374] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:19.046932 27480 raft_consensus.cc:374] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:19.046952 27478 raft_consensus.cc:729] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65658aa7935544b68919aab9c84dbb38, State: Initialized, Role: FOLLOWER
I20260924 01:07:19.047127 27480 raft_consensus.cc:729] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9ac384bf5f41df8b55eb5f76979ba4, State: Initialized, Role: FOLLOWER
I20260924 01:07:19.047766 27480 consensus_queue.cc:260] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.047740 27478 consensus_queue.cc:260] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.049400 27460 heartbeater.cc:503] Master 127.8.254.126:41837 was elected leader, sending a full tablet report...
I20260924 01:07:19.049685 27479 raft_consensus.cc:348] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.050103 27479 raft_consensus.cc:374] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:19.050239 27480 ts_tablet_manager.cc:1428] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 01:07:19.050232 27479 raft_consensus.cc:729] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502a8ff222034405ac3006bb36686784, State: Initialized, Role: FOLLOWER
I20260924 01:07:19.050838 27479 consensus_queue.cc:260] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.051748 27478 ts_tablet_manager.cc:1428] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260924 01:07:19.052692 27330 heartbeater.cc:503] Master 127.8.254.126:41837 was elected leader, sending a full tablet report...
I20260924 01:07:19.053946 27479 ts_tablet_manager.cc:1428] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
I20260924 01:07:19.081918 27484 raft_consensus.cc:482] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:07:19.082306 27484 raft_consensus.cc:504] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.084095 27484 leader_election.cc:288] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502a8ff222034405ac3006bb36686784 (127.8.254.66:36131), 65658aa7935544b68919aab9c84dbb38 (127.8.254.67:40163)
I20260924 01:07:19.093446 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.084665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:59045 (local address 127.8.254.67:40163)
0924 01:07:19.084858 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:19.084864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.085576 (+ 712us) server_negotiation.cc:407] Connection header received
0924 01:07:19.085766 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.085772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.085854 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.085963 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:19.086779 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.087910 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.090997 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.092279 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.092454 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.092520 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.093053 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.093182 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:07:19.093446 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.084659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:40073 (local address 127.8.254.66:36131)
0924 01:07:19.084858 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:19.084864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.085594 (+ 730us) server_negotiation.cc:407] Connection header received
0924 01:07:19.085766 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.085772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.085874 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.085963 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:19.086934 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.087954 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.091091 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.092296 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.092449 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.092520 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.093053 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.093182 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:07:19.093665 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.084503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:36131 (local address 127.8.254.65:40073)
0924 01:07:19.085484 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:19.085500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:19.085613 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:19.085997 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:19.086002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:19.086040 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:19.086691 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.086705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.088090 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.088095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.090775 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.090791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.092553 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.092559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.092885 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.092908 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 01:07:19.093006 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 01:07:19.093209 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":278,"spinlock_wait_cycles":6144,"thread_start_us":568,"threads_started":1}
I20260924 01:07:19.093765 27487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.084384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:40163 (local address 127.8.254.65:59045)
0924 01:07:19.085428 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:19.085452 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:19.085620 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:19.085997 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:19.086002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:19.086034 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:19.086605 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.086625 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.088089 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.088095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.090775 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:19.090791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.092532 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.092540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:19.092868 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.092908 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 01:07:19.093006 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 01:07:19.093209 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":340,"spinlock_wait_cycles":39040,"thread_start_us":649,"threads_started":1}
I20260924 01:07:19.094986 27285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f09232fab6a41e590a193d6bf7234ea" candidate_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502a8ff222034405ac3006bb36686784" is_pre_election: true
I20260924 01:07:19.094987 27415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f09232fab6a41e590a193d6bf7234ea" candidate_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65658aa7935544b68919aab9c84dbb38" is_pre_election: true
I20260924 01:07:19.095587 27415 raft_consensus.cc:2454] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9ac384bf5f41df8b55eb5f76979ba4 in term 0.
I20260924 01:07:19.095587 27285 raft_consensus.cc:2454] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9ac384bf5f41df8b55eb5f76979ba4 in term 0.
I20260924 01:07:19.096336 27088 leader_election.cc:302] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 502a8ff222034405ac3006bb36686784, 65658aa7935544b68919aab9c84dbb38, 7d9ac384bf5f41df8b55eb5f76979ba4; no voters:
I20260924 01:07:19.096803 27484 raft_consensus.cc:2781] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:07:19.096963 27484 raft_consensus.cc:482] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:07:19.097069 27484 raft_consensus.cc:3037] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:19.099370 27484 raft_consensus.cc:504] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.099973 27484 leader_election.cc:288] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [CANDIDATE]: Term 1 election: Requested vote from peers 502a8ff222034405ac3006bb36686784 (127.8.254.66:36131), 65658aa7935544b68919aab9c84dbb38 (127.8.254.67:40163)
I20260924 01:07:19.100459 27285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f09232fab6a41e590a193d6bf7234ea" candidate_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502a8ff222034405ac3006bb36686784"
I20260924 01:07:19.100478 27415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f09232fab6a41e590a193d6bf7234ea" candidate_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65658aa7935544b68919aab9c84dbb38"
I20260924 01:07:19.100656 27285 raft_consensus.cc:3037] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:19.100682 27415 raft_consensus.cc:3037] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:19.103477 27415 raft_consensus.cc:2454] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9ac384bf5f41df8b55eb5f76979ba4 in term 1.
I20260924 01:07:19.103515 27285 raft_consensus.cc:2454] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9ac384bf5f41df8b55eb5f76979ba4 in term 1.
I20260924 01:07:19.103900 27090 leader_election.cc:302] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65658aa7935544b68919aab9c84dbb38, 7d9ac384bf5f41df8b55eb5f76979ba4; no voters:
I20260924 01:07:19.104226 27484 raft_consensus.cc:2781] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:19.104696 27484 raft_consensus.cc:686] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9ac384bf5f41df8b55eb5f76979ba4, State: Running, Role: LEADER
I20260924 01:07:19.105239 27484 consensus_queue.cc:237] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } }
I20260924 01:07:19.109607 27013 catalog_manager.cc:5495] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d9ac384bf5f41df8b55eb5f76979ba4 (127.8.254.65). New cstate: current_term: 1 leader_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ac384bf5f41df8b55eb5f76979ba4" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 } health_report { overall_health: UNKNOWN } } }
I20260924 01:07:19.138127 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.133024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.8.254.65:41217)
0924 01:07:19.133201 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:19.133205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.133242 (+ 37us) server_negotiation.cc:407] Connection header received
0924 01:07:19.133445 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.133448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.133543 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.133681 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:07:19.134027 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.134790 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.136173 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.136572 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.136666 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.137132 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:07:19.137623 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:07:19.137733 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.137870 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.137932 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 01:07:19.146788 27415 raft_consensus.cc:1261] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 1 FOLLOWER]: Refusing update from remote peer 7d9ac384bf5f41df8b55eb5f76979ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:19.146828 27285 raft_consensus.cc:1261] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 1 FOLLOWER]: Refusing update from remote peer 7d9ac384bf5f41df8b55eb5f76979ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:19.147835 27489 consensus_queue.cc:1035] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502a8ff222034405ac3006bb36686784" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:19.148222 27484 consensus_queue.cc:1035] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65658aa7935544b68919aab9c84dbb38" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:19.159477 27492 mvcc.cc:205] Tried to move back new op lower bound from 7332708512333910016 to 7332708512178700288. Current Snapshot: MvccSnapshot[applied={T|T < 7332708512333910016}]
I20260924 01:07:19.174712 27011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:19.175004 27011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:19.179072 27011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b4b3b53d8d34bb4bffe4294f5f4044a: Sending DeleteTablet for 3 replicas of tablet 8f09232fab6a41e590a193d6bf7234ea
I20260924 01:07:19.180338 27265 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f09232fab6a41e590a193d6bf7234ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:19 UTC) from {username='slave'} at 127.0.0.1:55160
I20260924 01:07:19.180344 27134 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f09232fab6a41e590a193d6bf7234ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:19 UTC) from {username='slave'} at 127.0.0.1:57748
I20260924 01:07:19.181268 27503 tablet_replica.cc:330] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: stopping tablet replica
I20260924 01:07:19.181447 27395 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f09232fab6a41e590a193d6bf7234ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:19 UTC) from {username='slave'} at 127.0.0.1:57502
I20260924 01:07:19.181787 27503 raft_consensus.cc:2229] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:19.182339 27503 raft_consensus.cc:2258] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:19.183374 27502 tablet_replica.cc:330] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: stopping tablet replica
I20260924 01:07:19.183760 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27072
I20260924 01:07:19.183873 27502 raft_consensus.cc:2229] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:07:19.184528 27502 raft_consensus.cc:2258] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:19.186517 27504 tablet_replica.cc:330] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: stopping tablet replica
I20260924 01:07:19.186928 27504 raft_consensus.cc:2229] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:19.187431 27504 raft_consensus.cc:2258] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:19.189630 27504 ts_tablet_manager.cc:1905] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:19.189656 27502 ts_tablet_manager.cc:1905] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:19.193444 27503 ts_tablet_manager.cc:1905] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:19.196810 27504 ts_tablet_manager.cc:1918] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:19.197013 27504 log.cc:1196] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/8f09232fab6a41e590a193d6bf7234ea
I20260924 01:07:19.197620 27504 ts_tablet_manager.cc:1939] T 8f09232fab6a41e590a193d6bf7234ea P 65658aa7935544b68919aab9c84dbb38: Deleting consensus metadata
I20260924 01:07:19.199573 27000 catalog_manager.cc:4841] TS 65658aa7935544b68919aab9c84dbb38 (127.8.254.67:40163): tablet 8f09232fab6a41e590a193d6bf7234ea (table test-table [id=db6b1ff26ff64f10ad2cd12509c176dd]) successfully deleted
I20260924 01:07:19.200145 27502 ts_tablet_manager.cc:1918] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:19.200330 27502 log.cc:1196] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/8f09232fab6a41e590a193d6bf7234ea
I20260924 01:07:19.200743 27503 ts_tablet_manager.cc:1918] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:19.200901 27502 ts_tablet_manager.cc:1939] T 8f09232fab6a41e590a193d6bf7234ea P 7d9ac384bf5f41df8b55eb5f76979ba4: Deleting consensus metadata
I20260924 01:07:19.200924 27503 log.cc:1196] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/8f09232fab6a41e590a193d6bf7234ea
I20260924 01:07:19.201350 27503 ts_tablet_manager.cc:1939] T 8f09232fab6a41e590a193d6bf7234ea P 502a8ff222034405ac3006bb36686784: Deleting consensus metadata
I20260924 01:07:19.202790 26999 catalog_manager.cc:4841] TS 7d9ac384bf5f41df8b55eb5f76979ba4 (127.8.254.65:41217): tablet 8f09232fab6a41e590a193d6bf7234ea (table test-table [id=db6b1ff26ff64f10ad2cd12509c176dd]) successfully deleted
I20260924 01:07:19.203464 26998 catalog_manager.cc:4841] TS 502a8ff222034405ac3006bb36686784 (127.8.254.66:36131): tablet 8f09232fab6a41e590a193d6bf7234ea (table test-table [id=db6b1ff26ff64f10ad2cd12509c176dd]) successfully deleted
I20260924 01:07:19.204059 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.186993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.8.254.65:41217)
0924 01:07:19.187554 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:19.187563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.188123 (+ 560us) server_negotiation.cc:407] Connection header received
0924 01:07:19.188409 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.188414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.188529 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.188608 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:19.189302 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.190540 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.192726 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.193393 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.193491 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.202184 (+ 8693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:19.202219 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:19.202239 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:19.203099 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:19.203157 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:19.203178 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:19.203374 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.203545 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.203678 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397}
I20260924 01:07:19.204710 27194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:19.302016 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27072
I20260924 01:07:19.322479 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27203
I20260924 01:07:19.329510 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.323290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.8.254.66:36131)
0924 01:07:19.323512 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:19.323518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.323538 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:07:19.323729 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.323735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.323848 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.323966 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:19.324372 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.325112 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.326445 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.326849 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.326987 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.328092 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:19.328102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:19.328117 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:19.328718 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:19.328768 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:19.328789 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:19.328931 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.329067 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.329180 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 01:07:19.330263 27325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:19.420925 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27203
I20260924 01:07:19.438405 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27333
I20260924 01:07:19.445009 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:19.439307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.8.254.67:40163)
0924 01:07:19.439474 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:19.439479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:19.439496 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:07:19.439550 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:19.439554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:19.439688 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:19.439785 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:19.440125 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.440912 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.442148 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:19.442699 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:19.442796 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:19.443790 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:19.443799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:19.443814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:19.444402 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:19.444449 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:19.444468 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:19.444561 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:19.444675 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 01:07:19.444759 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 01:07:19.445672 27455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:19.539737 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27333
I20260924 01:07:19.559842 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 26982
I20260924 01:07:19.560989 27043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:19.671113 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 26982
2026-09-24T01:07:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2762 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:19 dist-test-slave-5c7l krb5kdc[27512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:19 dist-test-slave-5c7l krb5kdc[27512](info): set up 2 sockets
Sep 24 01:07:19 dist-test-slave-5c7l krb5kdc[27512](info): commencing operation
krb5kdc: starting...
W20260924 01:07:20.783486 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:20 dist-test-slave-5c7l krb5kdc[27512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:07:21.042585 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:42565
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:42565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:21.379492 27525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:21.379861 27525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:21.380016 27525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:21.387452 27525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:21.387599 27525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:21.387707 27525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:21.387811 27525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:21.387899 27525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:21.397703 27525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:42565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:21.399056 27525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:21.400341 27525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:21.412289 27530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:21.412338 27531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:21.412364 27533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:21.413729 27525 server_base.cc:1034] running on GCE node
I20260924 01:07:21.414521 27525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:21.415856 27525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:21.417085 27525 hybrid_clock.cc:648] HybridClock initialized: now 1790212041417037 us; error 70 us; skew 500 ppm
I20260924 01:07:21.417438 27525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:21 dist-test-slave-5c7l krb5kdc[27512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:21.420523 27525 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:07:21.422047 27525 webserver.cc:460] Webserver started at http://127.8.254.126:35117/ using document root <none> and password file <none>
I20260924 01:07:21.422780 27525 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:21.422926 27525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:21.423234 27525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:21.426144 27525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "ed61a79630a34bd09e5eb7bd3cc47b30"
format_stamp: "Formatted at 2026-09-24 01:07:21 on dist-test-slave-5c7l"
server_key: "fa169fcb727d60fd410737c1518ac223"
server_key_iv: "d0842a4ff9e7a9be616001b875154a50"
server_key_version: "encryptionkey@0"
I20260924 01:07:21.426981 27525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "ed61a79630a34bd09e5eb7bd3cc47b30"
format_stamp: "Formatted at 2026-09-24 01:07:21 on dist-test-slave-5c7l"
server_key: "fa169fcb727d60fd410737c1518ac223"
server_key_iv: "d0842a4ff9e7a9be616001b875154a50"
server_key_version: "encryptionkey@0"
I20260924 01:07:21.432096 27525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:07:21.435957 27540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:21.437307 27525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:07:21.437510 27525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "ed61a79630a34bd09e5eb7bd3cc47b30"
format_stamp: "Formatted at 2026-09-24 01:07:21 on dist-test-slave-5c7l"
server_key: "fa169fcb727d60fd410737c1518ac223"
server_key_iv: "d0842a4ff9e7a9be616001b875154a50"
server_key_version: "encryptionkey@0"
I20260924 01:07:21.437729 27525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:21.456918 27525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:21.461084 27525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:21.461381 27525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:21.480189 27525 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:42565
I20260924 01:07:21.480208 27591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:42565 every 8 connection(s)
I20260924 01:07:21.481632 27525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:21.485523 27592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:21.491604 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27525
I20260924 01:07:21.491786 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:21.492396 9209 external_mini_cluster.cc:1428] Setting key d03cb5e158574ad76b2d1deb7ba0e809
I20260924 01:07:21.501201 27592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30: Bootstrap starting.
I20260924 01:07:21.504429 27592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:21.505609 27592 log.cc:824] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30: Log is configured to *not* fsync() on all Append() calls
Sep 24 01:07:21 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:21.508805 27592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30: No bootstrap required, opened a new log
I20260924 01:07:21.517696 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:21.494459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.8.254.126:42565)
0924 01:07:21.495390 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:21.495408 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:21.495465 (+ 57us) server_negotiation.cc:407] Connection header received
0924 01:07:21.497766 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:21.497797 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:21.498316 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:21.498739 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:21.500388 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:21.501774 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:21.503333 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:21.504017 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:21.504218 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:21.508762 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:21.508778 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:21.508806 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:21.508861 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:21.512912 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:21.513692 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:21.513705 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:21.513714 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:21.513933 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:21.514345 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:21.514351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:21.514354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:21.515141 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:21.515453 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:21.515820 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 01:07:21.516293 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":418,"threads_started":1}
I20260924 01:07:21.520129 27592 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42565 } }
I20260924 01:07:21.520514 27592 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:21.520625 27592 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed61a79630a34bd09e5eb7bd3cc47b30, State: Initialized, Role: FOLLOWER
I20260924 01:07:21.521278 27592 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42565 } }
I20260924 01:07:21.521524 27592 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:21.521678 27592 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:21.521862 27592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:21.524103 27592 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42565 } }
I20260924 01:07:21.524602 27592 leader_election.cc:302] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed61a79630a34bd09e5eb7bd3cc47b30; no voters:
I20260924 01:07:21.525175 27592 leader_election.cc:288] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:21.525357 27597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:21.526853 27597 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [term 1 LEADER]: Becoming Leader. State: Replica: ed61a79630a34bd09e5eb7bd3cc47b30, State: Running, Role: LEADER
I20260924 01:07:21.527374 27597 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42565 } }
I20260924 01:07:21.527478 27592 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:21.531365 27598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42565 } } }
I20260924 01:07:21.531762 27598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:21.531322 27599 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed61a79630a34bd09e5eb7bd3cc47b30. Latest consensus state: current_term: 1 leader_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed61a79630a34bd09e5eb7bd3cc47b30" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42565 } } }
I20260924 01:07:21.533254 27599 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:21.534469 27607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:21.539983 27607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:21.549234 27607 catalog_manager.cc:1347] Generated new cluster ID: 3398517d687141358d4536b9a729d642
I20260924 01:07:21.549384 27607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:21.568558 27607 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:21.570005 27607 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:21.583879 27607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30: Generated new TSK 0
I20260924 01:07:21.584769 27607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:07:21.642197 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:42565
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:21.976755 27618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:21.977028 27618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:21.977231 27618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:21.984442 27618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:21.984552 27618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:21.984772 27618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:07:21.994805 27618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:21.996162 27618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:21.997462 27618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:22.011011 27626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:22.011049 27624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:22.011222 27623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:22.011910 27618 server_base.cc:1034] running on GCE node
I20260924 01:07:22.012524 27618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:22.013543 27618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:22.014803 27618 hybrid_clock.cc:648] HybridClock initialized: now 1790212042014759 us; error 58 us; skew 500 ppm
I20260924 01:07:22.015132 27618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:22 dist-test-slave-5c7l krb5kdc[27512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:22.018356 27618 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:07:22.019897 27618 webserver.cc:460] Webserver started at http://127.8.254.65:43909/ using document root <none> and password file <none>
I20260924 01:07:22.020530 27618 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:22.020628 27618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:22.020895 27618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:22.023847 27618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "a2ef2b773ed94cd0a2723446cd063485"
format_stamp: "Formatted at 2026-09-24 01:07:22 on dist-test-slave-5c7l"
server_key: "03a99b4e706299fd8f4d9e49c6afa8f7"
server_key_iv: "22b2bfd8819d9f9dfe1ad2ab9d90ca15"
server_key_version: "encryptionkey@0"
I20260924 01:07:22.024573 27618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "a2ef2b773ed94cd0a2723446cd063485"
format_stamp: "Formatted at 2026-09-24 01:07:22 on dist-test-slave-5c7l"
server_key: "03a99b4e706299fd8f4d9e49c6afa8f7"
server_key_iv: "22b2bfd8819d9f9dfe1ad2ab9d90ca15"
server_key_version: "encryptionkey@0"
I20260924 01:07:22.029839 27618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:22.033243 27633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:22.034896 27618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:07:22.035066 27618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "a2ef2b773ed94cd0a2723446cd063485"
format_stamp: "Formatted at 2026-09-24 01:07:22 on dist-test-slave-5c7l"
server_key: "03a99b4e706299fd8f4d9e49c6afa8f7"
server_key_iv: "22b2bfd8819d9f9dfe1ad2ab9d90ca15"
server_key_version: "encryptionkey@0"
I20260924 01:07:22.035276 27618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:22.052717 27618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:22.056816 27618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:22.057116 27618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:22.058094 27618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:22.059657 27618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:22.059775 27618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:22.059928 27618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:22.060000 27618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:22.093448 27618 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:37383
I20260924 01:07:22.093490 27745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:37383 every 8 connection(s)
I20260924 01:07:22.094904 27618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:07:22.097002 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27618
I20260924 01:07:22.097196 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:07:22.097730 9209 external_mini_cluster.cc:1428] Setting key 2983b1645a48b3d7a567b463ec8582dd
Sep 24 01:07:22 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:22.119148 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:22.097642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:40889 (local address 127.8.254.126:42565)
0924 01:07:22.098592 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:22.098599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:22.100305 (+ 1706us) server_negotiation.cc:407] Connection header received
0924 01:07:22.102171 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:22.102178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:22.102296 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:22.102410 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:22.104042 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.105268 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.106842 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.107594 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.107756 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:22.111893 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:22.111902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:22.111910 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:22.111955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:22.115697 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:22.116471 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:22.116480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:22.116487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:22.116651 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:22.117093 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:22.117101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:22.117108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:22.117500 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:22.117662 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:22.118380 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 01:07:22.118643 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":686,"threads_started":1}
I20260924 01:07:22.119773 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:22.098390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:42565 (local address 127.8.254.65:40889)
0924 01:07:22.100146 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:22.100191 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:22.101902 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:22.102643 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:22.102651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:22.103156 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:22.103870 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:22.103888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.105427 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.105431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:22.106705 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:22.106717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.107855 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.107879 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:22.108214 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:22.108964 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:22.108999 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:22.111655 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:22.115868 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:22.115874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:22.115886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:22.116313 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:22.116770 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:22.116773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:22.116775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:22.116958 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:22.117623 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:22.117629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:22.117952 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 01:07:22.118188 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 01:07:22.118629 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1469,"thread_start_us":1257,"threads_started":1}
I20260924 01:07:22.121461 27746 heartbeater.cc:347] Connected to a master server at 127.8.254.126:42565
I20260924 01:07:22.121824 27746 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:22.122851 27746 heartbeater.cc:511] Master 127.8.254.126:42565 requested a full tablet report, sending...
I20260924 01:07:22.125165 27557 ts_manager.cc:194] Registered new tserver with Master: a2ef2b773ed94cd0a2723446cd063485 (127.8.254.65:37383)
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:07:22.126904 27557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:40889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:07:22.158491 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:42565
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:22.493162 27752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:22.493489 27752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:22.493677 27752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:22.500967 27752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:22.501127 27752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:22.501314 27752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:07:22.511402 27752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:22.512729 27752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:22.514076 27752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:22.527794 27757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:22.527822 27760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:22.527854 27758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:22.528955 27752 server_base.cc:1034] running on GCE node
I20260924 01:07:22.529592 27752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:22.530702 27752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:22.531957 27752 hybrid_clock.cc:648] HybridClock initialized: now 1790212042531910 us; error 57 us; skew 500 ppm
I20260924 01:07:22.532300 27752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:22 dist-test-slave-5c7l krb5kdc[27512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:22.544829 27752 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:07:22.546597 27752 webserver.cc:460] Webserver started at http://127.8.254.66:39361/ using document root <none> and password file <none>
I20260924 01:07:22.547358 27752 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:22.547497 27752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:22.547837 27752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:22.551402 27752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "93109757f87f42eabc7cabf0a67719f6"
format_stamp: "Formatted at 2026-09-24 01:07:22 on dist-test-slave-5c7l"
server_key: "9140613e00e6d6f8ede4ad746e15a4f2"
server_key_iv: "16042acde196b5119f9ce587b8bab2f6"
server_key_version: "encryptionkey@0"
I20260924 01:07:22.552570 27752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "93109757f87f42eabc7cabf0a67719f6"
format_stamp: "Formatted at 2026-09-24 01:07:22 on dist-test-slave-5c7l"
server_key: "9140613e00e6d6f8ede4ad746e15a4f2"
server_key_iv: "16042acde196b5119f9ce587b8bab2f6"
server_key_version: "encryptionkey@0"
I20260924 01:07:22.558130 27752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 01:07:22.561772 27767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:22.563249 27752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:22.563447 27752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "93109757f87f42eabc7cabf0a67719f6"
format_stamp: "Formatted at 2026-09-24 01:07:22 on dist-test-slave-5c7l"
server_key: "9140613e00e6d6f8ede4ad746e15a4f2"
server_key_iv: "16042acde196b5119f9ce587b8bab2f6"
server_key_version: "encryptionkey@0"
I20260924 01:07:22.563719 27752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:22.578011 27752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:22.581950 27752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:22.582316 27752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:22.583391 27752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:22.584836 27752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:22.584939 27752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:22.585093 27752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:22.585172 27752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:22.617204 27752 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:44929
I20260924 01:07:22.617242 27879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:44929 every 8 connection(s)
I20260924 01:07:22.618631 27752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:07:22.623852 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27752
I20260924 01:07:22.624053 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:07:22.624521 9209 external_mini_cluster.cc:1428] Setting key bb6a4b142accfcd2c7ce875e443f8ed8
Sep 24 01:07:22 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:22.641561 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:22.621437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:53573 (local address 127.8.254.126:42565)
0924 01:07:22.621924 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:22.621931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:22.622978 (+ 1047us) server_negotiation.cc:407] Connection header received
0924 01:07:22.624995 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:22.625001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:22.625098 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:22.625208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:22.626967 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.628184 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.629885 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.630660 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.630801 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:22.634396 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:22.634406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:22.634413 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:22.634463 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:22.637480 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:22.638695 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:22.638701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:22.638703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:22.638847 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:22.639429 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:22.639434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:22.639436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:22.639873 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:22.639996 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:22.640963 (+ 967us) server_negotiation.cc:299] Negotiation successful
0924 01:07:22.641204 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260924 01:07:22.643047 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:22.621998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:42565 (local address 127.8.254.66:53573)
0924 01:07:22.622792 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:22.622832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:22.624700 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:22.625501 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:22.625510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:22.626033 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:22.626790 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:22.626808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.628334 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.628338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:22.629736 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:22.629748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:22.630871 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:22.630876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:22.631072 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:22.631796 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:22.631820 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:22.634073 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:22.637660 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:22.637672 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:22.637692 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:22.638424 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:22.638969 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:22.638977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:22.638983 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:22.639262 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:22.640006 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:22.640017 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:22.640516 (+ 499us) client_negotiation.cc:769] Sending connection context
0924 01:07:22.640851 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 01:07:22.641480 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":356,"threads_started":1}
I20260924 01:07:22.644716 27880 heartbeater.cc:347] Connected to a master server at 127.8.254.126:42565
I20260924 01:07:22.645074 27880 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:22.645831 27880 heartbeater.cc:511] Master 127.8.254.126:42565 requested a full tablet report, sending...
I20260924 01:07:22.647532 27557 ts_manager.cc:194] Registered new tserver with Master: 93109757f87f42eabc7cabf0a67719f6 (127.8.254.66:44929)
I20260924 01:07:22.648909 27557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:53573
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:07:22.686172 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:42565
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:23.017992 27886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:23.018368 27886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:23.018471 27886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:23.025729 27886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:23.025848 27886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:23.025964 27886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:07:23.036144 27886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:23.037436 27886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:23.038735 27886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:23.053027 27891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:23.053093 27892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:23.053119 27894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:23.053994 27886 server_base.cc:1034] running on GCE node
I20260924 01:07:23.054701 27886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:23.055706 27886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:23.056972 27886 hybrid_clock.cc:648] HybridClock initialized: now 1790212043056922 us; error 38 us; skew 500 ppm
I20260924 01:07:23.057341 27886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:23 dist-test-slave-5c7l krb5kdc[27512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212043, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:23.060819 27886 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:07:23.062548 27886 webserver.cc:460] Webserver started at http://127.8.254.67:36023/ using document root <none> and password file <none>
I20260924 01:07:23.063254 27886 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:23.063359 27886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:23.063633 27886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:23.066617 27886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "5a4f42ea740649a1aae3870e21bf42dd"
format_stamp: "Formatted at 2026-09-24 01:07:23 on dist-test-slave-5c7l"
server_key: "6336248fd821008ddc71fa4e19e02040"
server_key_iv: "8567ed0726250581a30aa3ec92cf4e53"
server_key_version: "encryptionkey@0"
I20260924 01:07:23.067423 27886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "5a4f42ea740649a1aae3870e21bf42dd"
format_stamp: "Formatted at 2026-09-24 01:07:23 on dist-test-slave-5c7l"
server_key: "6336248fd821008ddc71fa4e19e02040"
server_key_iv: "8567ed0726250581a30aa3ec92cf4e53"
server_key_version: "encryptionkey@0"
I20260924 01:07:23.072772 27886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:07:23.076333 27901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:23.077698 27886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:23.077872 27886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "5a4f42ea740649a1aae3870e21bf42dd"
format_stamp: "Formatted at 2026-09-24 01:07:23 on dist-test-slave-5c7l"
server_key: "6336248fd821008ddc71fa4e19e02040"
server_key_iv: "8567ed0726250581a30aa3ec92cf4e53"
server_key_version: "encryptionkey@0"
I20260924 01:07:23.078084 27886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:23.105422 27886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:23.109834 27886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:23.110167 27886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:23.111224 27886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:23.112646 27886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:23.112718 27886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:23.112841 27886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:23.112912 27886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:23.130178 27746 heartbeater.cc:503] Master 127.8.254.126:42565 was elected leader, sending a full tablet report...
I20260924 01:07:23.147235 27886 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:41677
I20260924 01:07:23.147265 28013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:41677 every 8 connection(s)
I20260924 01:07:23.148577 27886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:07:23.151469 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 27886
I20260924 01:07:23.151679 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:07:23.152161 9209 external_mini_cluster.cc:1428] Setting key 491c0ea5f20b2aa7f65bd06433ca0a6a
Sep 24 01:07:23 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212043, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:23.169417 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.151359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:39965 (local address 127.8.254.126:42565)
0924 01:07:23.151867 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.151875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.153152 (+ 1277us) server_negotiation.cc:407] Connection header received
0924 01:07:23.155378 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.155383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.155476 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.155593 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:23.157115 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.157866 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.159487 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.159893 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.160027 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.163496 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:23.163501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:23.163504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:23.163538 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:23.166074 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.167015 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.167021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.167024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.167192 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.167734 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.167739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.167744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.168118 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:23.168241 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.168938 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.169111 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
I20260924 01:07:23.170456 28017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.151967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:42565 (local address 127.8.254.67:39965)
0924 01:07:23.152748 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:23.152787 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:23.155073 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:23.155852 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:23.155861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:23.156373 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:23.156950 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.156968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.158022 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.158027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.159333 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.159345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.160118 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.160125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.160365 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.161019 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:23.161042 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:23.163283 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:23.166243 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:23.166254 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:23.166272 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:23.166828 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:23.167339 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:23.167345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:23.167348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:23.167598 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:23.168249 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:23.168260 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:23.168595 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 01:07:23.168824 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 01:07:23.169222 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":305,"threads_started":1}
I20260924 01:07:23.171842 28014 heartbeater.cc:347] Connected to a master server at 127.8.254.126:42565
I20260924 01:07:23.172173 28014 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:23.172896 28014 heartbeater.cc:511] Master 127.8.254.126:42565 requested a full tablet report, sending...
I20260924 01:07:23.174404 27557 ts_manager.cc:194] Registered new tserver with Master: 5a4f42ea740649a1aae3870e21bf42dd (127.8.254.67:41677)
I20260924 01:07:23.175223 27557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:39965
I20260924 01:07:23.180243 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:07:23.197162 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.184581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.8.254.126:42565)
0924 01:07:23.184870 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.184875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.185231 (+ 356us) server_negotiation.cc:407] Connection header received
0924 01:07:23.185390 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.185395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.185486 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.185573 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:23.187048 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.187787 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.189154 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.189545 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.189634 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.192240 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:23.192244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:23.192248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:23.192278 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:23.194785 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.195470 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.195473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.195476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.195580 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.195945 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.195948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.195950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.196277 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:23.196471 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.196797 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.196930 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260924 01:07:23.201303 27557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:07:23.208230 27557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:07:23.242017 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.221019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.8.254.66:44929)
0924 01:07:23.231876 (+ 10857us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.231889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.231928 (+ 39us) server_negotiation.cc:407] Connection header received
0924 01:07:23.232064 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.232077 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.232429 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.232681 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:23.233475 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.235641 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.239105 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.240795 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.240930 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.241260 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.241498 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.241669 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10577,"spinlock_wait_cycles":30592,"thread_start_us":10543,"threads_started":1}
I20260924 01:07:23.242067 28027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.220455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:44929 (local address 127.0.0.1:59268)
0924 01:07:23.231643 (+ 11188us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:23.231663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:23.231767 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:23.232687 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:23.232698 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:23.232737 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:23.233239 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.233255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.235918 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.235927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.238924 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.238939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.240997 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.241004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.241227 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.241289 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 01:07:23.241429 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 01:07:23.241629 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11076,"thread_start_us":10962,"threads_started":1}
I20260924 01:07:23.244200 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.220533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.8.254.65:37383)
0924 01:07:23.233467 (+ 12934us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.233478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.233515 (+ 37us) server_negotiation.cc:407] Connection header received
0924 01:07:23.233645 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.233657 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.233955 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.234276 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:23.237145 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.238799 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.241264 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.243033 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.243193 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.243456 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.243706 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.243876 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12737,"thread_start_us":12590,"threads_started":1}
I20260924 01:07:23.244205 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.220384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37383 (local address 127.0.0.1:52758)
0924 01:07:23.229918 (+ 9534us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:23.229971 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:23.230188 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:23.235850 (+ 5662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:23.235858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:23.235896 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:23.236577 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.236593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.238940 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.238945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.241076 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.241090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.243274 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.243280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.243499 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.243533 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:07:23.243637 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 01:07:23.243782 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9216,"mutex_wait_us":4,"thread_start_us":17480,"threads_started":1}
I20260924 01:07:23.247328 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.226014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:41677 (local address 127.0.0.1:49656)
0924 01:07:23.237641 (+ 11627us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:23.237661 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:23.237785 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:23.239545 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:23.239549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:23.239575 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:23.239987 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.239998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.242151 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.242157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.244602 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.244617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.246410 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.246417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.246600 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.246643 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 01:07:23.246752 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 01:07:23.246881 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11485,"mutex_wait_us":61,"thread_start_us":3431,"threads_started":1}
I20260924 01:07:23.246718 27815 tablet_service.cc:1467] Processing CreateTablet for tablet 41a053d0ac4b4bb7977eb4f21d08be0e (DEFAULT_TABLE table=test-table [id=5238fe5ad4aa494aa5f4ca8872e4aa75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:23.248296 27815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a053d0ac4b4bb7977eb4f21d08be0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:23.248363 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.226662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.8.254.67:41677)
0924 01:07:23.238595 (+ 11933us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.238604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.238646 (+ 42us) server_negotiation.cc:407] Connection header received
0924 01:07:23.238742 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.238751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.239043 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.239248 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:23.240199 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.241967 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.244809 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.245866 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.246004 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.246274 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.247830 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.248025 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11756,"thread_start_us":11555,"threads_started":1}
I20260924 01:07:23.248311 27681 tablet_service.cc:1467] Processing CreateTablet for tablet 41a053d0ac4b4bb7977eb4f21d08be0e (DEFAULT_TABLE table=test-table [id=5238fe5ad4aa494aa5f4ca8872e4aa75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:23.249697 27681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a053d0ac4b4bb7977eb4f21d08be0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:23.251977 27949 tablet_service.cc:1467] Processing CreateTablet for tablet 41a053d0ac4b4bb7977eb4f21d08be0e (DEFAULT_TABLE table=test-table [id=5238fe5ad4aa494aa5f4ca8872e4aa75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:23.253325 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a053d0ac4b4bb7977eb4f21d08be0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:23.259680 28033 tablet_bootstrap.cc:492] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Bootstrap starting.
I20260924 01:07:23.263020 28033 tablet_bootstrap.cc:654] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:23.264398 28033 log.cc:824] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:23.265822 28034 tablet_bootstrap.cc:492] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Bootstrap starting.
I20260924 01:07:23.267725 28033 tablet_bootstrap.cc:492] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: No bootstrap required, opened a new log
I20260924 01:07:23.267988 28033 ts_tablet_manager.cc:1397] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 01:07:23.269385 28034 tablet_bootstrap.cc:654] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:23.270670 28034 log.cc:824] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:23.270999 28036 tablet_bootstrap.cc:492] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Bootstrap starting.
I20260924 01:07:23.274420 28036 tablet_bootstrap.cc:654] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:23.274895 28034 tablet_bootstrap.cc:492] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: No bootstrap required, opened a new log
I20260924 01:07:23.275171 28034 ts_tablet_manager.cc:1397] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 01:07:23.275748 28036 log.cc:824] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:23.278323 28036 tablet_bootstrap.cc:492] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: No bootstrap required, opened a new log
I20260924 01:07:23.278581 28036 ts_tablet_manager.cc:1397] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260924 01:07:23.279436 28033 raft_consensus.cc:348] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.279913 28033 raft_consensus.cc:374] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:23.280016 28033 raft_consensus.cc:729] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93109757f87f42eabc7cabf0a67719f6, State: Initialized, Role: FOLLOWER
I20260924 01:07:23.280663 28033 consensus_queue.cc:260] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.281996 27880 heartbeater.cc:503] Master 127.8.254.126:42565 was elected leader, sending a full tablet report...
I20260924 01:07:23.282618 28033 ts_tablet_manager.cc:1428] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 01:07:23.285100 28034 raft_consensus.cc:348] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.285467 28034 raft_consensus.cc:374] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:23.285588 28034 raft_consensus.cc:729] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ef2b773ed94cd0a2723446cd063485, State: Initialized, Role: FOLLOWER
I20260924 01:07:23.286203 28034 consensus_queue.cc:260] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.288021 28034 ts_tablet_manager.cc:1428] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 01:07:23.288303 28036 raft_consensus.cc:348] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.288772 28036 raft_consensus.cc:374] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:23.288879 28036 raft_consensus.cc:729] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4f42ea740649a1aae3870e21bf42dd, State: Initialized, Role: FOLLOWER
I20260924 01:07:23.289619 28036 consensus_queue.cc:260] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.290935 28014 heartbeater.cc:503] Master 127.8.254.126:42565 was elected leader, sending a full tablet report...
I20260924 01:07:23.291633 28036 ts_tablet_manager.cc:1428] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 01:07:23.339128 28039 raft_consensus.cc:482] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:07:23.339509 28039 raft_consensus.cc:504] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.341461 28039 leader_election.cc:288] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2ef2b773ed94cd0a2723446cd063485 (127.8.254.65:37383), 5a4f42ea740649a1aae3870e21bf42dd (127.8.254.67:41677)
W20260924 01:07:23.347340 27747 tablet.cc:2369] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:23.348419 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.341869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:37435 (local address 127.8.254.65:37383)
0924 01:07:23.342136 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.342146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.342635 (+ 489us) server_negotiation.cc:407] Connection header received
0924 01:07:23.342909 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.342913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.342975 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.343078 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:23.343629 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.344330 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.346311 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.347142 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.347363 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.347445 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.347960 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.348140 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 01:07:23.348784 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.342141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:42237 (local address 127.8.254.67:41677)
0924 01:07:23.342382 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.342388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.343014 (+ 626us) server_negotiation.cc:407] Connection header received
0924 01:07:23.343226 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.343231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.343290 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.343391 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:23.343940 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.344660 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.346492 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.347561 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.347742 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.347840 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.348386 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.348530 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 01:07:23.348879 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.341740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:41677 (local address 127.8.254.66:42237)
0924 01:07:23.342891 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:23.342905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:23.343061 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:23.343414 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:23.343418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:23.343447 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:23.343787 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.343795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.344813 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.344818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.346339 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.346352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.347921 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.347969 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.348187 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.348223 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 01:07:23.348332 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 01:07:23.348485 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":3,"thread_start_us":440,"threads_started":1}
I20260924 01:07:23.349328 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.341703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:37383 (local address 127.8.254.66:37435)
0924 01:07:23.342496 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:23.342545 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:23.342740 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:23.343097 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:23.343102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:23.343128 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:23.343475 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.343484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.344472 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.344480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.346110 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:23.346124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.347367 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.347373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:23.347623 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.347662 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 01:07:23.348709 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0924 01:07:23.348862 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":47,"thread_start_us":877,"threads_started":1}
I20260924 01:07:23.350160 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a053d0ac4b4bb7977eb4f21d08be0e" candidate_uuid: "93109757f87f42eabc7cabf0a67719f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4f42ea740649a1aae3870e21bf42dd" is_pre_election: true
I20260924 01:07:23.350862 27969 raft_consensus.cc:2454] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93109757f87f42eabc7cabf0a67719f6 in term 0.
I20260924 01:07:23.350731 27701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a053d0ac4b4bb7977eb4f21d08be0e" candidate_uuid: "93109757f87f42eabc7cabf0a67719f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ef2b773ed94cd0a2723446cd063485" is_pre_election: true
I20260924 01:07:23.351329 27701 raft_consensus.cc:2454] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93109757f87f42eabc7cabf0a67719f6 in term 0.
I20260924 01:07:23.351589 27769 leader_election.cc:302] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4f42ea740649a1aae3870e21bf42dd, 93109757f87f42eabc7cabf0a67719f6; no voters:
I20260924 01:07:23.352025 28039 raft_consensus.cc:2781] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:07:23.352185 28039 raft_consensus.cc:482] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:07:23.352308 28039 raft_consensus.cc:3037] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:23.354744 28039 raft_consensus.cc:504] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.355429 28039 leader_election.cc:288] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [CANDIDATE]: Term 1 election: Requested vote from peers a2ef2b773ed94cd0a2723446cd063485 (127.8.254.65:37383), 5a4f42ea740649a1aae3870e21bf42dd (127.8.254.67:41677)
I20260924 01:07:23.355855 27701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a053d0ac4b4bb7977eb4f21d08be0e" candidate_uuid: "93109757f87f42eabc7cabf0a67719f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ef2b773ed94cd0a2723446cd063485"
I20260924 01:07:23.356014 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a053d0ac4b4bb7977eb4f21d08be0e" candidate_uuid: "93109757f87f42eabc7cabf0a67719f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4f42ea740649a1aae3870e21bf42dd"
I20260924 01:07:23.356077 27701 raft_consensus.cc:3037] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:23.356222 27969 raft_consensus.cc:3037] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:23.359400 27701 raft_consensus.cc:2454] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93109757f87f42eabc7cabf0a67719f6 in term 1.
I20260924 01:07:23.359440 27969 raft_consensus.cc:2454] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93109757f87f42eabc7cabf0a67719f6 in term 1.
I20260924 01:07:23.359894 27771 leader_election.cc:302] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a4f42ea740649a1aae3870e21bf42dd, 93109757f87f42eabc7cabf0a67719f6, a2ef2b773ed94cd0a2723446cd063485; no voters:
I20260924 01:07:23.360276 28039 raft_consensus.cc:2781] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:23.360718 28039 raft_consensus.cc:686] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 1 LEADER]: Becoming Leader. State: Replica: 93109757f87f42eabc7cabf0a67719f6, State: Running, Role: LEADER
I20260924 01:07:23.361199 28039 consensus_queue.cc:237] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } }
I20260924 01:07:23.366078 27556 catalog_manager.cc:5495] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93109757f87f42eabc7cabf0a67719f6 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "93109757f87f42eabc7cabf0a67719f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93109757f87f42eabc7cabf0a67719f6" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 } health_report { overall_health: UNKNOWN } } }
W20260924 01:07:23.370880 27881 tablet.cc:2369] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:23.392747 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.387704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.8.254.66:44929)
0924 01:07:23.387888 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.387892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.387976 (+ 84us) server_negotiation.cc:407] Connection header received
0924 01:07:23.388137 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.388141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.388220 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.388349 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:07:23.388688 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.389417 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.390841 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.391217 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.391322 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.391777 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:07:23.392256 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:07:23.392385 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.392493 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.392531 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 01:07:23.400305 28015 tablet.cc:2369] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:23.401202 27969 raft_consensus.cc:1261] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 1 FOLLOWER]: Refusing update from remote peer 93109757f87f42eabc7cabf0a67719f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:23.401269 27701 raft_consensus.cc:1261] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 1 FOLLOWER]: Refusing update from remote peer 93109757f87f42eabc7cabf0a67719f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:23.402179 28039 consensus_queue.cc:1035] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4f42ea740649a1aae3870e21bf42dd" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:23.402522 28044 consensus_queue.cc:1035] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2ef2b773ed94cd0a2723446cd063485" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:23.413465 28047 mvcc.cc:205] Tried to move back new op lower bound from 7332708529759944704 to 7332708529611816960. Current Snapshot: MvccSnapshot[applied={T|T < 7332708529759944704}]
I20260924 01:07:23.428054 27556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:23.428383 27556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:23.432077 27556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed61a79630a34bd09e5eb7bd3cc47b30: Sending DeleteTablet for 3 replicas of tablet 41a053d0ac4b4bb7977eb4f21d08be0e
I20260924 01:07:23.433303 27815 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a053d0ac4b4bb7977eb4f21d08be0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:23 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:59268
I20260924 01:07:23.433342 27681 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a053d0ac4b4bb7977eb4f21d08be0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:23 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:52758
I20260924 01:07:23.433785 27949 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a053d0ac4b4bb7977eb4f21d08be0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:23 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:49656
I20260924 01:07:23.436519 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27618
I20260924 01:07:23.443284 28058 tablet_replica.cc:330] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: stopping tablet replica
I20260924 01:07:23.443667 28056 tablet_replica.cc:330] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: stopping tablet replica
I20260924 01:07:23.443914 28058 raft_consensus.cc:2229] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:23.444324 28056 raft_consensus.cc:2229] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:07:23.444466 28057 tablet_replica.cc:330] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: stopping tablet replica
I20260924 01:07:23.444576 28058 raft_consensus.cc:2258] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:23.444852 28057 raft_consensus.cc:2229] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:23.445230 28056 raft_consensus.cc:2258] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:23.445822 28057 raft_consensus.cc:2258] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:23.447719 28058 ts_tablet_manager.cc:1905] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:23.447896 28056 ts_tablet_manager.cc:1905] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:23.447937 28057 ts_tablet_manager.cc:1905] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:23.455094 28056 ts_tablet_manager.cc:1918] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:23.455094 28057 ts_tablet_manager.cc:1918] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:23.455113 28058 ts_tablet_manager.cc:1918] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:23.455273 28057 log.cc:1196] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/41a053d0ac4b4bb7977eb4f21d08be0e
I20260924 01:07:23.455304 28058 log.cc:1196] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/41a053d0ac4b4bb7977eb4f21d08be0e
I20260924 01:07:23.455498 28056 log.cc:1196] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/41a053d0ac4b4bb7977eb4f21d08be0e
I20260924 01:07:23.455895 28057 ts_tablet_manager.cc:1939] T 41a053d0ac4b4bb7977eb4f21d08be0e P a2ef2b773ed94cd0a2723446cd063485: Deleting consensus metadata
I20260924 01:07:23.455936 28058 ts_tablet_manager.cc:1939] T 41a053d0ac4b4bb7977eb4f21d08be0e P 5a4f42ea740649a1aae3870e21bf42dd: Deleting consensus metadata
I20260924 01:07:23.456166 28056 ts_tablet_manager.cc:1939] T 41a053d0ac4b4bb7977eb4f21d08be0e P 93109757f87f42eabc7cabf0a67719f6: Deleting consensus metadata
I20260924 01:07:23.457863 27543 catalog_manager.cc:4841] TS 93109757f87f42eabc7cabf0a67719f6 (127.8.254.66:44929): tablet 41a053d0ac4b4bb7977eb4f21d08be0e (table test-table [id=5238fe5ad4aa494aa5f4ca8872e4aa75]) successfully deleted
I20260924 01:07:23.458120 27544 catalog_manager.cc:4841] TS a2ef2b773ed94cd0a2723446cd063485 (127.8.254.65:37383): tablet 41a053d0ac4b4bb7977eb4f21d08be0e (table test-table [id=5238fe5ad4aa494aa5f4ca8872e4aa75]) successfully deleted
I20260924 01:07:23.458122 27542 catalog_manager.cc:4841] TS 5a4f42ea740649a1aae3870e21bf42dd (127.8.254.67:41677): tablet 41a053d0ac4b4bb7977eb4f21d08be0e (table test-table [id=5238fe5ad4aa494aa5f4ca8872e4aa75]) successfully deleted
Sep 24 01:07:23 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:07:23.472783 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.438941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.8.254.65:37383)
0924 01:07:23.439194 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.439203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.457019 (+ 17816us) server_negotiation.cc:407] Connection header received
0924 01:07:23.457547 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.457553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.457636 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.457719 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:23.459540 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.460253 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.461588 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.461988 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.462126 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.466624 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:23.466638 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:23.466653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:23.466689 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:23.469794 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.470625 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.470633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.470636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.470773 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.471191 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.471196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.471198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.471564 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:23.471905 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.472195 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.472420 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 01:07:23.473467 27741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:23.596287 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27618
I20260924 01:07:23.621371 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27752
Sep 24 01:07:23 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:07:23.642513 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.622153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.8.254.66:44929)
0924 01:07:23.622405 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.622414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.622439 (+ 25us) server_negotiation.cc:407] Connection header received
0924 01:07:23.622696 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.622702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.622832 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.622966 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:23.624650 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.625879 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.627878 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.628541 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.628693 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.634631 (+ 5938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:23.634647 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:23.634667 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:23.634711 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:23.639013 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.640005 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.640015 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.640020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.640195 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.640643 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.640649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.640653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.641134 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:23.641455 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.641839 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.642093 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 01:07:23.643226 27875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:23.758134 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27752
I20260924 01:07:23.778386 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27886
Sep 24 01:07:23 dist-test-slave-5c7l krb5kdc[27512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:07:23.793768 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:23.779177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.8.254.67:41677)
0924 01:07:23.779369 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:23.779373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:23.779392 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:07:23.779563 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:23.779567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:23.779632 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:23.779720 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:23.781064 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.781804 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.783369 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:23.783762 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:23.783878 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:23.788384 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:23.788398 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:23.788414 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:23.788449 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:23.791172 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.791834 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.791841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.791843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.791969 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:23.792390 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:23.792393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:23.792395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:23.792732 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:23.792952 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:23.793212 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 01:07:23.793387 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:07:23.794507 28009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:23.883051 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27886
I20260924 01:07:23.900904 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 27525
I20260924 01:07:23.901971 27587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:24.013887 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 27525
2026-09-24T01:07:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4343 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T01:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:24Z Disabled control of system clock
I20260924 01:07:24.092446 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45861
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 01:07:24.423743 28067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:24.424073 28067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:24.424223 28067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:24.431371 28067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:24.431556 28067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:24.431627 28067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:24.431730 28067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:24.431828 28067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:24.441434 28067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:24.442834 28067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:24.444619 28067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:24.456581 28073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:24.456633 28075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:24.456609 28072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:24.457621 28067 server_base.cc:1034] running on GCE node
I20260924 01:07:24.458505 28067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:24.459993 28067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:24.461246 28067 hybrid_clock.cc:648] HybridClock initialized: now 1790212044461183 us; error 57 us; skew 500 ppm
I20260924 01:07:24.461616 28067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:24.464226 28067 webserver.cc:460] Webserver started at http://127.8.254.126:37703/ using document root <none> and password file <none>
I20260924 01:07:24.464962 28067 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:24.465095 28067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:24.465428 28067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:24.468561 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "97d789e4da0f405883d1be8ec91e1b47"
format_stamp: "Formatted at 2026-09-24 01:07:24 on dist-test-slave-5c7l"
server_key: "388262edce0a47540919da31d609406f"
server_key_iv: "c4275f12c4d3600f4502541b1624de85"
server_key_version: "encryptionkey@0"
I20260924 01:07:24.469411 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "97d789e4da0f405883d1be8ec91e1b47"
format_stamp: "Formatted at 2026-09-24 01:07:24 on dist-test-slave-5c7l"
server_key: "388262edce0a47540919da31d609406f"
server_key_iv: "c4275f12c4d3600f4502541b1624de85"
server_key_version: "encryptionkey@0"
I20260924 01:07:24.474851 28067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:07:24.478667 28081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:24.480100 28067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:24.480296 28067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "97d789e4da0f405883d1be8ec91e1b47"
format_stamp: "Formatted at 2026-09-24 01:07:24 on dist-test-slave-5c7l"
server_key: "388262edce0a47540919da31d609406f"
server_key_iv: "c4275f12c4d3600f4502541b1624de85"
server_key_version: "encryptionkey@0"
I20260924 01:07:24.480527 28067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:24.502883 28067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:24.503757 28067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:24.504056 28067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:24.522622 28067 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45861
I20260924 01:07:24.522646 28133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45861 every 8 connection(s)
I20260924 01:07:24.523993 28067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:24.525501 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28067
I20260924 01:07:24.525678 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:24.526300 9209 external_mini_cluster.cc:1428] Setting key 12a848c7e4206d7e2333f01bfc236a45
I20260924 01:07:24.528678 28134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:24.542284 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:24.528252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.8.254.126:45861)
0924 01:07:24.529343 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:24.529357 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:24.529406 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:07:24.531686 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:24.531714 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:24.532326 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:24.532790 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:24.533334 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:24.535428 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:24.536846 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:24.537871 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:24.538065 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:24.539309 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:24.539324 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:24.539344 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:24.540088 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:24.540202 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:24.540239 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:24.540367 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:24.540587 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 01:07:24.540936 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":350,"threads_started":1}
I20260924 01:07:24.543355 28134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47: Bootstrap starting.
I20260924 01:07:24.546824 28134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:24.547838 28134 log.cc:824] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:24.550695 28134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47: No bootstrap required, opened a new log
I20260924 01:07:24.560012 28134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d789e4da0f405883d1be8ec91e1b47" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45861 } }
I20260924 01:07:24.560566 28134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:24.560670 28134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d789e4da0f405883d1be8ec91e1b47, State: Initialized, Role: FOLLOWER
I20260924 01:07:24.561362 28134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d789e4da0f405883d1be8ec91e1b47" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45861 } }
I20260924 01:07:24.561640 28134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:24.561789 28134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:24.561971 28134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:24.565443 28134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d789e4da0f405883d1be8ec91e1b47" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45861 } }
I20260924 01:07:24.566087 28134 leader_election.cc:302] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97d789e4da0f405883d1be8ec91e1b47; no voters:
I20260924 01:07:24.566740 28134 leader_election.cc:288] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:24.566910 28139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:24.567669 28139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [term 1 LEADER]: Becoming Leader. State: Replica: 97d789e4da0f405883d1be8ec91e1b47, State: Running, Role: LEADER
I20260924 01:07:24.568197 28139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d789e4da0f405883d1be8ec91e1b47" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45861 } }
I20260924 01:07:24.568884 28134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:24.571218 28140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97d789e4da0f405883d1be8ec91e1b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d789e4da0f405883d1be8ec91e1b47" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45861 } } }
I20260924 01:07:24.571595 28140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:24.571727 28141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97d789e4da0f405883d1be8ec91e1b47. Latest consensus state: current_term: 1 leader_uuid: "97d789e4da0f405883d1be8ec91e1b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d789e4da0f405883d1be8ec91e1b47" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45861 } } }
I20260924 01:07:24.572161 28141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:24.572916 28148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:24.578917 28148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:24.588055 28148 catalog_manager.cc:1347] Generated new cluster ID: ba53da1982ec4252bb1f4076817cc341
I20260924 01:07:24.588172 28148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:24.603855 28148 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:24.605736 28148 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:24.614921 28148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47: Generated new TSK 0
I20260924 01:07:24.616009 28148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:24.627079 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45861
--builtin_ntp_servers=127.8.254.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 01:07:24.960949 28158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:24.961243 28158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:24.961418 28158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:24.968848 28158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:24.968991 28158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:24.969178 28158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:07:24.979173 28158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:24.980592 28158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:24.982648 28158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:24.996294 28166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:24.996239 28164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:24.996347 28163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:24.999975 28158 server_base.cc:1034] running on GCE node
I20260924 01:07:25.000761 28158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:25.001811 28158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:25.003055 28158 hybrid_clock.cc:648] HybridClock initialized: now 1790212045003008 us; error 48 us; skew 500 ppm
I20260924 01:07:25.003409 28158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:25.005867 28158 webserver.cc:460] Webserver started at http://127.8.254.65:37141/ using document root <none> and password file <none>
I20260924 01:07:25.006680 28158 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:25.006820 28158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:25.007156 28158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:25.010167 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "6792797d864248e98a48953c845ad3bb"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "73e3f3f195f2ed1af3b5750d76ea6b82"
server_key_iv: "b3eb954ffc381fc760c7d91bf3c3a1be"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.011121 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "6792797d864248e98a48953c845ad3bb"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "73e3f3f195f2ed1af3b5750d76ea6b82"
server_key_iv: "b3eb954ffc381fc760c7d91bf3c3a1be"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.016602 28158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:07:25.020326 28172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.021842 28158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:25.022027 28158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "6792797d864248e98a48953c845ad3bb"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "73e3f3f195f2ed1af3b5750d76ea6b82"
server_key_iv: "b3eb954ffc381fc760c7d91bf3c3a1be"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.022276 28158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:25.034957 28158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:25.035800 28158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:25.036098 28158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:25.037032 28158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:25.038556 28158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:25.038668 28158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.038789 28158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:25.038887 28158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.072345 28158 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:38801
I20260924 01:07:25.072435 28284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:38801 every 8 connection(s)
I20260924 01:07:25.073761 28158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:07:25.082471 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28158
I20260924 01:07:25.082640 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:07:25.083078 9209 external_mini_cluster.cc:1428] Setting key 59c9d9dbbfd8c730d99f5f275cc041a8
I20260924 01:07:25.086237 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45861
--builtin_ntp_servers=127.8.254.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 01:07:25.086875 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:25.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:49231 (local address 127.8.254.126:45861)
0924 01:07:25.076943 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:25.076948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:25.078117 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 01:07:25.079993 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:25.079999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:25.080088 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:25.080207 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:25.081272 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.081984 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.083777 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.084394 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.084512 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:25.086055 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:25.086060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:25.086064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:25.086083 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:25.086126 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:25.086138 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:25.086268 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:25.086542 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 01:07:25.086620 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":250,"threads_started":1}
I20260924 01:07:25.088357 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:25.077044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45861 (local address 127.8.254.65:49231)
0924 01:07:25.077950 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:25.078002 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:25.079717 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:25.080460 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:25.080468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:25.080556 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:25.081105 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:25.081120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.082151 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.082155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:25.083627 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:25.083640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.084618 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.084623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:25.084943 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:25.085703 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:25.085707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:25.085759 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:25.085859 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:25.086286 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:25.086298 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:25.086308 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 01:07:25.086507 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 01:07:25.086929 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
I20260924 01:07:25.089921 28285 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45861
I20260924 01:07:25.090274 28285 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:25.091162 28285 heartbeater.cc:511] Master 127.8.254.126:45861 requested a full tablet report, sending...
I20260924 01:07:25.093389 28098 ts_manager.cc:194] Registered new tserver with Master: 6792797d864248e98a48953c845ad3bb (127.8.254.65:38801)
I20260924 01:07:25.095549 28098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.65:49231
W20260924 01:07:25.418985 28289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:25.419253 28289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:25.419337 28289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:25.426860 28289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:25.426970 28289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:25.427152 28289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:07:25.436244 28289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:25.437543 28289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:25.439399 28289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:25.452565 28294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:25.452491 28295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:25.452524 28297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:25.453753 28289 server_base.cc:1034] running on GCE node
I20260924 01:07:25.454419 28289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:25.455381 28289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:25.456583 28289 hybrid_clock.cc:648] HybridClock initialized: now 1790212045456527 us; error 47 us; skew 500 ppm
I20260924 01:07:25.456898 28289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:25.459160 28289 webserver.cc:460] Webserver started at http://127.8.254.66:45967/ using document root <none> and password file <none>
I20260924 01:07:25.459837 28289 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:25.459950 28289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:25.460270 28289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:25.463140 28289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "21fb2abf96fb4537a5475ad1b7489235"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "d0f4740f199956eefa13f162d9cda503"
server_key_iv: "7a16e85d781563b900b033cf9731d4fb"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.463919 28289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "21fb2abf96fb4537a5475ad1b7489235"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "d0f4740f199956eefa13f162d9cda503"
server_key_iv: "7a16e85d781563b900b033cf9731d4fb"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.469024 28289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 01:07:25.472482 28303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.473781 28289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 01:07:25.473951 28289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "21fb2abf96fb4537a5475ad1b7489235"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "d0f4740f199956eefa13f162d9cda503"
server_key_iv: "7a16e85d781563b900b033cf9731d4fb"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.474170 28289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:25.498975 28289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:25.499809 28289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:25.500129 28289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:25.501140 28289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:25.502645 28289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:25.502729 28289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.502859 28289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:25.502956 28289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.537714 28289 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:41489
I20260924 01:07:25.537770 28415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:41489 every 8 connection(s)
I20260924 01:07:25.539129 28289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:07:25.542475 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28289
I20260924 01:07:25.542645 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:07:25.543107 9209 external_mini_cluster.cc:1428] Setting key fade5e2533b37cc4d039db48f3e78f29
I20260924 01:07:25.545936 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45861
--builtin_ntp_servers=127.8.254.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 01:07:25.553566 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:25.541975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:38931 (local address 127.8.254.126:45861)
0924 01:07:25.542257 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:25.542265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:25.543891 (+ 1626us) server_negotiation.cc:407] Connection header received
0924 01:07:25.545716 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:25.545725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:25.545812 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:25.545925 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:25.547198 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.548502 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.550132 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.550879 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.551028 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:25.552549 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:25.552558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:25.552566 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:25.552595 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:25.552640 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:25.552658 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:25.552764 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:25.553010 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 01:07:25.553128 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 01:07:25.554818 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:25.542589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45861 (local address 127.8.254.66:38931)
0924 01:07:25.543711 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:25.543767 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:25.545405 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:25.546242 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:25.546256 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:25.546369 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:25.547019 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:25.547038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.548670 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.548675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:25.549978 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:25.549991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:25.551182 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:25.551188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:25.551414 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:25.552197 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:25.552200 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:25.552243 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:25.552337 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:25.552770 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:25.552779 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:25.552785 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 01:07:25.552965 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 01:07:25.553296 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":347,"threads_started":1}
I20260924 01:07:25.556463 28416 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45861
I20260924 01:07:25.556787 28416 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:25.557588 28416 heartbeater.cc:511] Master 127.8.254.126:45861 requested a full tablet report, sending...
I20260924 01:07:25.559082 28098 ts_manager.cc:194] Registered new tserver with Master: 21fb2abf96fb4537a5475ad1b7489235 (127.8.254.66:41489)
I20260924 01:07:25.560334 28098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.66:38931
W20260924 01:07:25.881577 28419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:25.881848 28419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:25.881934 28419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:25.889343 28419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:25.889444 28419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:25.889547 28419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:07:25.898473 28419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:25.899684 28419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:25.901388 28419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:25.914112 28425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:25.914132 28424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:25.914259 28427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:25.915187 28419 server_base.cc:1034] running on GCE node
I20260924 01:07:25.915709 28419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:25.916613 28419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:25.917807 28419 hybrid_clock.cc:648] HybridClock initialized: now 1790212045917750 us; error 50 us; skew 500 ppm
I20260924 01:07:25.918123 28419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:25.920293 28419 webserver.cc:460] Webserver started at http://127.8.254.67:39333/ using document root <none> and password file <none>
I20260924 01:07:25.920961 28419 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:25.921077 28419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:25.921408 28419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:25.924327 28419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "dfd6acdbeb8648f697dc157c0bdb06ff"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "67cfbf992f0619eec93b634d32554a4d"
server_key_iv: "15c7566ed546b29f237343adabe437d7"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.925141 28419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "dfd6acdbeb8648f697dc157c0bdb06ff"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "67cfbf992f0619eec93b634d32554a4d"
server_key_iv: "15c7566ed546b29f237343adabe437d7"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.930159 28419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 01:07:25.933561 28433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.934957 28419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:25.935138 28419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "dfd6acdbeb8648f697dc157c0bdb06ff"
format_stamp: "Formatted at 2026-09-24 01:07:25 on dist-test-slave-5c7l"
server_key: "67cfbf992f0619eec93b634d32554a4d"
server_key_iv: "15c7566ed546b29f237343adabe437d7"
server_key_version: "encryptionkey@0"
I20260924 01:07:25.935382 28419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:25.957526 28419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:25.958396 28419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:25.958706 28419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:25.959740 28419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:25.961189 28419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:25.961302 28419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.961440 28419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:25.961539 28419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:25.995484 28419 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:37901
I20260924 01:07:25.995576 28545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:37901 every 8 connection(s)
I20260924 01:07:25.996841 28419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:07:26.003757 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28419
I20260924 01:07:26.003953 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:07:26.004675 9209 external_mini_cluster.cc:1428] Setting key 4de595b3052c33c4e3114967187f6067
I20260924 01:07:26.010145 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:25.999438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:41625 (local address 127.8.254.126:45861)
0924 01:07:25.999619 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:25.999623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.001029 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 01:07:26.002944 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.002948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.003018 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.003111 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:26.004607 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.005419 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.007601 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.008050 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.008149 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.009484 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:26.009489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:26.009492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:26.009510 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:26.009541 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:26.009553 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:26.009625 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.009831 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.009911 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:07:26.011209 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:25.999924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:45861 (local address 127.8.254.67:41625)
0924 01:07:26.000832 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:26.000880 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:26.002643 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:26.003345 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:26.003359 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:26.003495 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:26.004409 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.004436 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.005584 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.005589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.007445 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.007462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.008240 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.008245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.008457 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.009185 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:26.009188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:26.009230 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:26.009315 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:26.009643 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:26.009651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:26.009657 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 01:07:26.009803 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 01:07:26.010129 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
I20260924 01:07:26.012564 28546 heartbeater.cc:347] Connected to a master server at 127.8.254.126:45861
I20260924 01:07:26.012867 28546 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:26.013541 28546 heartbeater.cc:511] Master 127.8.254.126:45861 requested a full tablet report, sending...
I20260924 01:07:26.014917 28098 ts_manager.cc:194] Registered new tserver with Master: dfd6acdbeb8648f697dc157c0bdb06ff (127.8.254.67:37901)
I20260924 01:07:26.015720 28098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.67:41625
I20260924 01:07:26.019917 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:07:26.029757 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.023981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.8.254.126:45861)
0924 01:07:26.024151 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.024156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.024625 (+ 469us) server_negotiation.cc:407] Connection header received
0924 01:07:26.024757 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.024761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.024831 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.024913 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:26.025431 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.026183 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.027636 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.028018 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.028115 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.029145 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:26.029150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:26.029153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:26.029172 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:26.029202 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:26.029214 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:26.029318 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.029497 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.029549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:07:26.033648 28098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:07:26.040681 28098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:07:26.075376 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.053109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.8.254.65:38801)
0924 01:07:26.064490 (+ 11381us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.064499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.064532 (+ 33us) server_negotiation.cc:407] Connection header received
0924 01:07:26.064650 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.064661 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.064987 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.065239 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:26.066561 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.068424 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.071639 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.073412 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.073542 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.073835 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.074707 (+ 872us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.074942 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11228,"thread_start_us":11097,"threads_started":1}
I20260924 01:07:26.077656 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.052690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:38801 (local address 127.0.0.1:50732)
0924 01:07:26.063776 (+ 11086us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:26.063805 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:26.063958 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:26.065634 (+ 1676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:26.065646 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:26.065689 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:26.066276 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.066294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.068581 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.068586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.071391 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.071402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.073975 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.073982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.074401 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.074467 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 01:07:26.076014 (+ 1547us) client_negotiation.cc:240] Negotiation successful
0924 01:07:26.077273 (+ 1259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10916,"mutex_wait_us":1386,"thread_start_us":10487,"threads_started":1}
I20260924 01:07:26.078053 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.064378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.8.254.67:37901)
0924 01:07:26.070024 (+ 5646us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.070030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.070056 (+ 26us) server_negotiation.cc:407] Connection header received
0924 01:07:26.070171 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.070177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.070429 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.070634 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:26.071282 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.072739 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.075382 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.076818 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.076941 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.077180 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.077394 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.077624 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5433,"thread_start_us":5268,"threads_started":1}
I20260924 01:07:26.077775 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.064588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:37901 (local address 127.0.0.1:60172)
0924 01:07:26.068039 (+ 3451us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:26.068060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:26.068163 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:26.070643 (+ 2480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:26.070648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:26.070676 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:26.071082 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.071093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.072918 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.072927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.075156 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.075171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.077030 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.077037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.077223 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.077255 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:07:26.077339 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 01:07:26.077462 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3340,"mutex_wait_us":23,"thread_start_us":3636,"threads_started":1}
I20260924 01:07:26.082054 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.052681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41489 (local address 127.0.0.1:58904)
0924 01:07:26.062945 (+ 10264us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:26.063006 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:26.063265 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:26.070178 (+ 6913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:26.070183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:26.070227 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:26.070710 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.070726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.073978 (+ 3252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.073982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.077026 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.077038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.081274 (+ 4236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.081281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.081451 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.081486 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:07:26.081576 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 01:07:26.081710 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9876,"mutex_wait_us":45,"thread_start_us":9699,"threads_started":1}
I20260924 01:07:26.082177 28220 tablet_service.cc:1467] Processing CreateTablet for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce (DEFAULT_TABLE table=test-table [id=6669b041ec9543dba043b5808b39bee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:26.083624 28220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:26.084412 28481 tablet_service.cc:1467] Processing CreateTablet for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce (DEFAULT_TABLE table=test-table [id=6669b041ec9543dba043b5808b39bee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:26.085767 28481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:26.091243 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.052907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.8.254.66:41489)
0924 01:07:26.069514 (+ 16607us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.069526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.069557 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:07:26.069666 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.069674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.069945 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.070156 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:26.070945 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.073803 (+ 2858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.077897 (+ 4094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.080605 (+ 2708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.080740 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.080986 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.081666 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.081849 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16436,"thread_start_us":16340,"threads_started":1}
I20260924 01:07:26.095448 28351 tablet_service.cc:1467] Processing CreateTablet for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce (DEFAULT_TABLE table=test-table [id=6669b041ec9543dba043b5808b39bee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:26.096261 28564 tablet_bootstrap.cc:492] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Bootstrap starting.
I20260924 01:07:26.096982 28351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:26.098485 28285 heartbeater.cc:503] Master 127.8.254.126:45861 was elected leader, sending a full tablet report...
I20260924 01:07:26.101402 28564 tablet_bootstrap.cc:654] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:26.102340 28565 tablet_bootstrap.cc:492] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Bootstrap starting.
I20260924 01:07:26.102825 28564 log.cc:824] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:26.107486 28565 tablet_bootstrap.cc:654] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:26.108781 28565 log.cc:824] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:26.108992 28564 tablet_bootstrap.cc:492] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: No bootstrap required, opened a new log
I20260924 01:07:26.109316 28564 ts_tablet_manager.cc:1397] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260924 01:07:26.112027 28568 tablet_bootstrap.cc:492] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Bootstrap starting.
I20260924 01:07:26.113193 28565 tablet_bootstrap.cc:492] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: No bootstrap required, opened a new log
I20260924 01:07:26.113469 28565 ts_tablet_manager.cc:1397] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 01:07:26.115248 28568 tablet_bootstrap.cc:654] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:26.116186 28568 log.cc:824] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:26.118708 28568 tablet_bootstrap.cc:492] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: No bootstrap required, opened a new log
I20260924 01:07:26.118960 28568 ts_tablet_manager.cc:1397] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 01:07:26.120453 28564 raft_consensus.cc:348] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.121016 28564 raft_consensus.cc:374] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:26.121125 28564 raft_consensus.cc:729] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6792797d864248e98a48953c845ad3bb, State: Initialized, Role: FOLLOWER
I20260924 01:07:26.121748 28564 consensus_queue.cc:260] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.123602 28564 ts_tablet_manager.cc:1428] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260924 01:07:26.123826 28565 raft_consensus.cc:348] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.124231 28565 raft_consensus.cc:374] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:26.124346 28565 raft_consensus.cc:729] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd6acdbeb8648f697dc157c0bdb06ff, State: Initialized, Role: FOLLOWER
I20260924 01:07:26.124914 28565 consensus_queue.cc:260] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.126011 28546 heartbeater.cc:503] Master 127.8.254.126:45861 was elected leader, sending a full tablet report...
I20260924 01:07:26.126711 28565 ts_tablet_manager.cc:1428] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260924 01:07:26.128890 28568 raft_consensus.cc:348] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.129232 28568 raft_consensus.cc:374] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:26.129366 28568 raft_consensus.cc:729] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fb2abf96fb4537a5475ad1b7489235, State: Initialized, Role: FOLLOWER
I20260924 01:07:26.129871 28568 consensus_queue.cc:260] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.130913 28416 heartbeater.cc:503] Master 127.8.254.126:45861 was elected leader, sending a full tablet report...
I20260924 01:07:26.131352 28568 ts_tablet_manager.cc:1428] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 01:07:26.191641 28570 raft_consensus.cc:482] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:07:26.191952 28570 raft_consensus.cc:504] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.193861 28570 leader_election.cc:288] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21fb2abf96fb4537a5475ad1b7489235 (127.8.254.66:41489), dfd6acdbeb8648f697dc157c0bdb06ff (127.8.254.67:37901)
I20260924 01:07:26.200444 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.194310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:42665 (local address 127.8.254.66:41489)
0924 01:07:26.194534 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.194542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.195201 (+ 659us) server_negotiation.cc:407] Connection header received
0924 01:07:26.195480 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.195486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.195565 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.195664 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:26.196254 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.196949 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.198823 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.199542 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.199729 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.199811 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.200066 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.200193 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:07:26.200541 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.194308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41489 (local address 127.8.254.65:42665)
0924 01:07:26.195090 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:26.195114 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:26.195325 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:26.195691 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:26.195696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:26.195720 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:26.196104 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.196113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.197089 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.197095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.198625 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.198637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.199729 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.199736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.199899 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.199923 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 01:07:26.200030 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 01:07:26.200161 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"spinlock_wait_cycles":34560,"thread_start_us":414,"threads_started":1}
I20260924 01:07:26.200719 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.194608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:37821 (local address 127.8.254.67:37901)
0924 01:07:26.194823 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.194829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.195761 (+ 932us) server_negotiation.cc:407] Connection header received
0924 01:07:26.195962 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.195968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.196035 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.196127 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:26.196643 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.197336 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.199092 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.199867 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.200038 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.200108 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.200368 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.200494 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 01:07:26.200961 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.194332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:37901 (local address 127.8.254.65:37821)
0924 01:07:26.195656 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:26.195670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:26.195824 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:26.196157 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:26.196161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:26.196184 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:26.196502 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.196510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.197479 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.197487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.198915 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:26.198928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.200063 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.200069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:26.200215 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.200237 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 01:07:26.200341 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 01:07:26.200474 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":1019,"threads_started":1}
I20260924 01:07:26.201887 28371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe91b3ddcc4fefaa20d8a9ed1fd0ce" candidate_uuid: "6792797d864248e98a48953c845ad3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fb2abf96fb4537a5475ad1b7489235" is_pre_election: true
I20260924 01:07:26.201907 28501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe91b3ddcc4fefaa20d8a9ed1fd0ce" candidate_uuid: "6792797d864248e98a48953c845ad3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" is_pre_election: true
I20260924 01:07:26.202515 28371 raft_consensus.cc:2454] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6792797d864248e98a48953c845ad3bb in term 0.
I20260924 01:07:26.202515 28501 raft_consensus.cc:2454] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6792797d864248e98a48953c845ad3bb in term 0.
I20260924 01:07:26.203249 28176 leader_election.cc:302] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21fb2abf96fb4537a5475ad1b7489235, 6792797d864248e98a48953c845ad3bb, dfd6acdbeb8648f697dc157c0bdb06ff; no voters:
I20260924 01:07:26.203655 28570 raft_consensus.cc:2781] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:07:26.203819 28570 raft_consensus.cc:482] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:07:26.203933 28570 raft_consensus.cc:3037] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:26.206169 28570 raft_consensus.cc:504] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.206852 28570 leader_election.cc:288] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [CANDIDATE]: Term 1 election: Requested vote from peers 21fb2abf96fb4537a5475ad1b7489235 (127.8.254.66:41489), dfd6acdbeb8648f697dc157c0bdb06ff (127.8.254.67:37901)
I20260924 01:07:26.207427 28501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe91b3ddcc4fefaa20d8a9ed1fd0ce" candidate_uuid: "6792797d864248e98a48953c845ad3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff"
I20260924 01:07:26.207414 28371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbe91b3ddcc4fefaa20d8a9ed1fd0ce" candidate_uuid: "6792797d864248e98a48953c845ad3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fb2abf96fb4537a5475ad1b7489235"
I20260924 01:07:26.207640 28371 raft_consensus.cc:3037] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:26.207640 28501 raft_consensus.cc:3037] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:26.210965 28371 raft_consensus.cc:2454] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6792797d864248e98a48953c845ad3bb in term 1.
I20260924 01:07:26.210965 28501 raft_consensus.cc:2454] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6792797d864248e98a48953c845ad3bb in term 1.
I20260924 01:07:26.211444 28176 leader_election.cc:302] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21fb2abf96fb4537a5475ad1b7489235, 6792797d864248e98a48953c845ad3bb, dfd6acdbeb8648f697dc157c0bdb06ff; no voters:
I20260924 01:07:26.211781 28570 raft_consensus.cc:2781] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:26.212229 28570 raft_consensus.cc:686] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 1 LEADER]: Becoming Leader. State: Replica: 6792797d864248e98a48953c845ad3bb, State: Running, Role: LEADER
I20260924 01:07:26.212710 28570 consensus_queue.cc:237] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } }
I20260924 01:07:26.217411 28098 catalog_manager.cc:5495] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6792797d864248e98a48953c845ad3bb (127.8.254.65). New cstate: current_term: 1 leader_uuid: "6792797d864248e98a48953c845ad3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6792797d864248e98a48953c845ad3bb" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 38801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 } health_report { overall_health: UNKNOWN } } }
I20260924 01:07:26.236444 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.231225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.8.254.65:38801)
0924 01:07:26.231467 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.231471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.231490 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:07:26.231604 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.231608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.231693 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.231820 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:07:26.232135 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.232835 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.234321 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.234715 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.234808 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.235280 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:07:26.235784 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:07:26.235939 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.236078 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.236146 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 01:07:26.245009 28371 raft_consensus.cc:1261] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 1 FOLLOWER]: Refusing update from remote peer 6792797d864248e98a48953c845ad3bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:26.245010 28501 raft_consensus.cc:1261] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 1 FOLLOWER]: Refusing update from remote peer 6792797d864248e98a48953c845ad3bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:26.246064 28575 consensus_queue.cc:1035] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd6acdbeb8648f697dc157c0bdb06ff" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:26.246521 28570 consensus_queue.cc:1035] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21fb2abf96fb4537a5475ad1b7489235" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 01:07:26.248634 28547 tablet.cc:2369] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:26.259565 28579 mvcc.cc:205] Tried to move back new op lower bound from 7332708541409038336 to 7332708541291118592. Current Snapshot: MvccSnapshot[applied={T|T < 7332708541409038336}]
I20260924 01:07:26.272572 28097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:26.272881 28097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:26.276680 28097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97d789e4da0f405883d1be8ec91e1b47: Sending DeleteTablet for 3 replicas of tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce
I20260924 01:07:26.277901 28351 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:26 UTC) from {username='slave'} at 127.0.0.1:58904
I20260924 01:07:26.278316 28481 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:26 UTC) from {username='slave'} at 127.0.0.1:60172
I20260924 01:07:26.278577 28220 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:26 UTC) from {username='slave'} at 127.0.0.1:50732
I20260924 01:07:26.280000 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28158
I20260924 01:07:26.289480 28590 tablet_replica.cc:330] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: stopping tablet replica
I20260924 01:07:26.290120 28590 raft_consensus.cc:2229] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:07:26.291124 28590 raft_consensus.cc:2258] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:26.294906 28590 ts_tablet_manager.cc:1905] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:26.302904 28590 ts_tablet_manager.cc:1918] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:26.303216 28590 log.cc:1196] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/dbbe91b3ddcc4fefaa20d8a9ed1fd0ce
I20260924 01:07:26.303916 28590 ts_tablet_manager.cc:1939] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 6792797d864248e98a48953c845ad3bb: Deleting consensus metadata
I20260924 01:07:26.306077 28082 catalog_manager.cc:4841] TS 6792797d864248e98a48953c845ad3bb (127.8.254.65:38801): tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce (table test-table [id=6669b041ec9543dba043b5808b39bee1]) successfully deleted
I20260924 01:07:26.307883 28588 tablet_replica.cc:330] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: stopping tablet replica
I20260924 01:07:26.308576 28588 raft_consensus.cc:2229] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:26.309269 28589 tablet_replica.cc:330] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: stopping tablet replica
I20260924 01:07:26.309662 28589 raft_consensus.cc:2229] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:26.310156 28589 raft_consensus.cc:2258] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:26.310922 28588 raft_consensus.cc:2258] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:26.312723 28588 ts_tablet_manager.cc:1905] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:26.315907 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.283120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.8.254.65:38801)
0924 01:07:26.283391 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.283399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.306714 (+ 23315us) server_negotiation.cc:407] Connection header received
0924 01:07:26.307060 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.307066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.307172 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.307294 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:26.307967 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.309185 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.311462 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.312115 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.312221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.313986 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:26.313999 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:26.314022 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:26.314888 (+ 866us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:26.314949 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:26.314979 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:26.315067 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.315345 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.315457 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 01:07:26.317015 28280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:26.318481 28589 ts_tablet_manager.cc:1905] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:26.319321 28588 ts_tablet_manager.cc:1918] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:26.319561 28588 log.cc:1196] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/dbbe91b3ddcc4fefaa20d8a9ed1fd0ce
I20260924 01:07:26.320182 28588 ts_tablet_manager.cc:1939] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P 21fb2abf96fb4537a5475ad1b7489235: Deleting consensus metadata
I20260924 01:07:26.322184 28085 catalog_manager.cc:4841] TS 21fb2abf96fb4537a5475ad1b7489235 (127.8.254.66:41489): tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce (table test-table [id=6669b041ec9543dba043b5808b39bee1]) successfully deleted
I20260924 01:07:26.325304 28589 ts_tablet_manager.cc:1918] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:26.325472 28589 log.cc:1196] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/dbbe91b3ddcc4fefaa20d8a9ed1fd0ce
I20260924 01:07:26.325966 28589 ts_tablet_manager.cc:1939] T dbbe91b3ddcc4fefaa20d8a9ed1fd0ce P dfd6acdbeb8648f697dc157c0bdb06ff: Deleting consensus metadata
I20260924 01:07:26.327594 28082 catalog_manager.cc:4841] TS dfd6acdbeb8648f697dc157c0bdb06ff (127.8.254.67:37901): tablet dbbe91b3ddcc4fefaa20d8a9ed1fd0ce (table test-table [id=6669b041ec9543dba043b5808b39bee1]) successfully deleted
I20260924 01:07:26.416855 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28158
I20260924 01:07:26.436023 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28289
I20260924 01:07:26.442996 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.436929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.8.254.66:41489)
0924 01:07:26.437101 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.437105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.437124 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:07:26.437228 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.437252 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.437345 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.437436 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:26.437804 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.438662 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.439914 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.440381 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.440475 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.441461 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:26.441469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:26.441484 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:26.442098 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:26.442146 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:26.442166 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:26.442340 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.442536 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.442624 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:07:26.443706 28411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:26.532519 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28289
I20260924 01:07:26.554105 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28419
I20260924 01:07:26.561267 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:26.554938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.8.254.67:37901)
0924 01:07:26.555131 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:26.555139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:26.555231 (+ 92us) server_negotiation.cc:407] Connection header received
0924 01:07:26.555453 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:26.555459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:26.555554 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:26.555664 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:26.556111 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.556870 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.558189 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:26.558689 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:26.558833 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:26.559908 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:26.559918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:26.559935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:26.560507 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:26.560555 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:26.560574 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:26.560699 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:26.560844 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 01:07:26.560967 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:07:26.561933 28541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:26.652572 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28419
I20260924 01:07:26.669818 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28067
I20260924 01:07:26.671036 28129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:26.782027 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28067
2026-09-24T01:07:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2767 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:26 dist-test-slave-5c7l krb5kdc[28598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:26 dist-test-slave-5c7l krb5kdc[28598](info): set up 2 sockets
Sep 24 01:07:26 dist-test-slave-5c7l krb5kdc[28598](info): commencing operation
krb5kdc: starting...
W20260924 01:07:27.903656 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:28 dist-test-slave-5c7l krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:07:28.165249 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:41789
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:41789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:28.504089 28611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:28.504381 28611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:28.504474 28611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:28.511713 28611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:28.511821 28611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:28.511868 28611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:28.511924 28611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:28.511968 28611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:28.522040 28611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:41789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:28.523432 28611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:28.524794 28611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:28.537099 28616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:28.537269 28619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:28.537132 28617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:28.538511 28611 server_base.cc:1034] running on GCE node
I20260924 01:07:28.539462 28611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:28.540977 28611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:28.542296 28611 hybrid_clock.cc:648] HybridClock initialized: now 1790212048542220 us; error 57 us; skew 500 ppm
I20260924 01:07:28.542681 28611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:28 dist-test-slave-5c7l krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212048, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:28.553442 28611 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:07:28.555365 28611 webserver.cc:460] Webserver started at http://127.8.254.126:42591/ using document root <none> and password file <none>
I20260924 01:07:28.556139 28611 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:28.556300 28611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:28.556633 28611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:28.559973 28611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "5d28e10f60644c2584123662a51dcca8"
format_stamp: "Formatted at 2026-09-24 01:07:28 on dist-test-slave-5c7l"
server_key: "27b5fce0b695c3b7fbb151fadae2e552"
server_key_iv: "43e9a6bf41871b6b9fd88f0a001c7297"
server_key_version: "encryptionkey@0"
I20260924 01:07:28.560783 28611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "5d28e10f60644c2584123662a51dcca8"
format_stamp: "Formatted at 2026-09-24 01:07:28 on dist-test-slave-5c7l"
server_key: "27b5fce0b695c3b7fbb151fadae2e552"
server_key_iv: "43e9a6bf41871b6b9fd88f0a001c7297"
server_key_version: "encryptionkey@0"
I20260924 01:07:28.566409 28611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:07:28.570128 28626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:28.571614 28611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:28.571812 28611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "5d28e10f60644c2584123662a51dcca8"
format_stamp: "Formatted at 2026-09-24 01:07:28 on dist-test-slave-5c7l"
server_key: "27b5fce0b695c3b7fbb151fadae2e552"
server_key_iv: "43e9a6bf41871b6b9fd88f0a001c7297"
server_key_version: "encryptionkey@0"
I20260924 01:07:28.572043 28611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:28.597827 28611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:28.601873 28611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:28.602164 28611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:28.621157 28611 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:41789
I20260924 01:07:28.621181 28677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:41789 every 8 connection(s)
I20260924 01:07:28.622597 28611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:28.626471 28678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:28.630951 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28611
I20260924 01:07:28.631134 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:28.631676 9209 external_mini_cluster.cc:1428] Setting key 0d9fd6ca9cbfe99dd19b7bd0f0c8cf78
I20260924 01:07:28.638316 28678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8: Bootstrap starting.
I20260924 01:07:28.642067 28678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:28.643271 28678 log.cc:824] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:28.646166 28678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8: No bootstrap required, opened a new log
Sep 24 01:07:28 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:28.656318 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:28.633653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.8.254.126:41789)
0924 01:07:28.634516 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:28.634527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:28.634568 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:07:28.636721 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:28.636747 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:28.637145 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:28.637494 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:28.639394 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:28.640752 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:28.642701 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:28.643451 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:28.643661 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:28.648449 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:28.648458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:28.648477 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:28.648528 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:28.651846 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:28.652793 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:28.652800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:28.652811 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:28.652989 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:28.653458 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:28.653461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:28.653463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:28.654059 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:28.654381 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:28.654795 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 01:07:28.655235 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":387,"threads_started":1}
I20260924 01:07:28.656150 28678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d28e10f60644c2584123662a51dcca8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41789 } }
I20260924 01:07:28.656572 28678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:28.656755 28678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d28e10f60644c2584123662a51dcca8, State: Initialized, Role: FOLLOWER
I20260924 01:07:28.657405 28678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d28e10f60644c2584123662a51dcca8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41789 } }
I20260924 01:07:28.657676 28678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:28.657841 28678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:28.658033 28678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:28.660668 28678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d28e10f60644c2584123662a51dcca8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41789 } }
I20260924 01:07:28.661185 28678 leader_election.cc:302] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d28e10f60644c2584123662a51dcca8; no voters:
I20260924 01:07:28.661796 28678 leader_election.cc:288] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:28.661945 28683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:28.662743 28683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [term 1 LEADER]: Becoming Leader. State: Replica: 5d28e10f60644c2584123662a51dcca8, State: Running, Role: LEADER
I20260924 01:07:28.663321 28683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d28e10f60644c2584123662a51dcca8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41789 } }
I20260924 01:07:28.663990 28678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:28.666846 28685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d28e10f60644c2584123662a51dcca8. Latest consensus state: current_term: 1 leader_uuid: "5d28e10f60644c2584123662a51dcca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d28e10f60644c2584123662a51dcca8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41789 } } }
I20260924 01:07:28.666930 28684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d28e10f60644c2584123662a51dcca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d28e10f60644c2584123662a51dcca8" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 41789 } } }
I20260924 01:07:28.667259 28685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:28.669077 28684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:28.669694 28691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:28.674337 28691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:28.683607 28691 catalog_manager.cc:1347] Generated new cluster ID: b3b0967c85a24acfa5087324dd4c324a
I20260924 01:07:28.683736 28691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:28.699779 28691 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:28.701145 28691 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:28.708312 28691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8: Generated new TSK 0
I20260924 01:07:28.709369 28691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.254.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
Entry for principal kudu/127.8.254.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab.
I20260924 01:07:28.781340 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41789
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:29.122401 28704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:29.122682 28704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:29.122835 28704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:29.130103 28704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:29.130263 28704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:29.130491 28704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:07:29.140605 28704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.65.keytab
--principal=kudu/127.8.254.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:29.141963 28704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:29.143362 28704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:29.156891 28710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:29.157140 28709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:29.157980 28712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:29.158326 28704 server_base.cc:1034] running on GCE node
I20260924 01:07:29.158985 28704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:29.159991 28704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:29.161201 28704 hybrid_clock.cc:648] HybridClock initialized: now 1790212049161135 us; error 54 us; skew 500 ppm
I20260924 01:07:29.161490 28704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:29 dist-test-slave-5c7l krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212049, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:29.164785 28704 init.cc:377] Logged in from keytab as kudu/127.8.254.65@KRBTEST.COM (short username kudu)
I20260924 01:07:29.166342 28704 webserver.cc:460] Webserver started at http://127.8.254.65:36667/ using document root <none> and password file <none>
I20260924 01:07:29.167007 28704 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:29.167104 28704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:29.167410 28704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:29.170637 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "d3d73afd424b4132b7c58d7a9bc04413"
format_stamp: "Formatted at 2026-09-24 01:07:29 on dist-test-slave-5c7l"
server_key: "fe2827df5414c1a3495a54b1262f74d6"
server_key_iv: "8eb15de137ad1577c5bf1a8375dbdfd9"
server_key_version: "encryptionkey@0"
I20260924 01:07:29.171399 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "d3d73afd424b4132b7c58d7a9bc04413"
format_stamp: "Formatted at 2026-09-24 01:07:29 on dist-test-slave-5c7l"
server_key: "fe2827df5414c1a3495a54b1262f74d6"
server_key_iv: "8eb15de137ad1577c5bf1a8375dbdfd9"
server_key_version: "encryptionkey@0"
I20260924 01:07:29.176678 28704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:07:29.180125 28719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:29.181613 28704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:29.181841 28704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "d3d73afd424b4132b7c58d7a9bc04413"
format_stamp: "Formatted at 2026-09-24 01:07:29 on dist-test-slave-5c7l"
server_key: "fe2827df5414c1a3495a54b1262f74d6"
server_key_iv: "8eb15de137ad1577c5bf1a8375dbdfd9"
server_key_version: "encryptionkey@0"
I20260924 01:07:29.182053 28704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:29.206537 28704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:29.210989 28704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:29.211299 28704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:29.212275 28704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:29.213580 28704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:29.213670 28704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:29.213842 28704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:29.213914 28704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:29.248466 28704 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:34545
I20260924 01:07:29.248564 28831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:34545 every 8 connection(s)
I20260924 01:07:29.249902 28704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:07:29.250481 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28704
I20260924 01:07:29.250685 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:07:29.251299 9209 external_mini_cluster.cc:1428] Setting key d4020df57e3eeb8963707e9b0c055efc
Sep 24 01:07:29 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212049, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.65@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:29.277251 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:29.253000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:37075 (local address 127.8.254.126:41789)
0924 01:07:29.253528 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:29.253535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:29.254711 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 01:07:29.256657 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:29.256663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:29.256757 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:29.256880 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:29.259602 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.260963 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.263485 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.264004 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.264147 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:29.268121 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:29.268131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:29.268138 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:29.268191 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:29.272220 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:29.273301 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:29.273309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:29.273316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:29.273474 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:29.274056 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:29.274064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:29.274070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:29.274511 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:29.274661 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:29.276496 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0924 01:07:29.276751 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":247,"threads_started":1}
I20260924 01:07:29.279409 28835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:29.253688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41789 (local address 127.8.254.65:37075)
0924 01:07:29.254561 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:29.254600 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:29.256372 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:29.257141 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:29.257150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:29.258248 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:29.259358 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:29.259388 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.261175 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.261180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:29.263300 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:29.263319 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.264251 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.264262 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:29.264553 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:29.265395 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:29.265432 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:29.267806 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:29.272404 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:29.272416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:29.272439 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:29.273108 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:29.273637 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:29.273645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:29.273652 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:29.273917 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:29.274757 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:29.274768 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:29.275369 (+ 601us) client_negotiation.cc:769] Sending connection context
0924 01:07:29.275691 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 01:07:29.277806 (+ 2115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":337,"threads_started":1}
I20260924 01:07:29.283092 28832 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41789
I20260924 01:07:29.283476 28832 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.254.66@KRBTEST.COM; defaulting to no policy
I20260924 01:07:29.284286 28832 heartbeater.cc:511] Master 127.8.254.126:41789 requested a full tablet report, sending...
I20260924 01:07:29.286726 28643 ts_manager.cc:194] Registered new tserver with Master: d3d73afd424b4132b7c58d7a9bc04413 (127.8.254.65:34545)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.66@KRBTEST.COM" created.
I20260924 01:07:29.288539 28643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.65@KRBTEST.COM'} at 127.8.254.65:37075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
Entry for principal kudu/127.8.254.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab.
I20260924 01:07:29.321847 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41789
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:29.664238 28838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:29.664582 28838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:29.664760 28838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:29.672420 28838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:29.672580 28838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:29.672765 28838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:07:29.683099 28838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.66.keytab
--principal=kudu/127.8.254.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:29.684499 28838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:29.686128 28838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:29.700155 28843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:29.700114 28846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:29.700206 28844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:29.708055 28838 server_base.cc:1034] running on GCE node
I20260924 01:07:29.708791 28838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:29.709874 28838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:29.711104 28838 hybrid_clock.cc:648] HybridClock initialized: now 1790212049711030 us; error 75 us; skew 500 ppm
I20260924 01:07:29.711431 28838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:29 dist-test-slave-5c7l krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212049, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:29.714677 28838 init.cc:377] Logged in from keytab as kudu/127.8.254.66@KRBTEST.COM (short username kudu)
I20260924 01:07:29.716331 28838 webserver.cc:460] Webserver started at http://127.8.254.66:33557/ using document root <none> and password file <none>
I20260924 01:07:29.717074 28838 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:29.717224 28838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:29.717554 28838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:29.720552 28838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1"
format_stamp: "Formatted at 2026-09-24 01:07:29 on dist-test-slave-5c7l"
server_key: "f4a3c695f0fd411bdc01a40bd3646360"
server_key_iv: "993a271ce289afe89713f0553a719c40"
server_key_version: "encryptionkey@0"
I20260924 01:07:29.721359 28838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1"
format_stamp: "Formatted at 2026-09-24 01:07:29 on dist-test-slave-5c7l"
server_key: "f4a3c695f0fd411bdc01a40bd3646360"
server_key_iv: "993a271ce289afe89713f0553a719c40"
server_key_version: "encryptionkey@0"
I20260924 01:07:29.726769 28838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 01:07:29.730232 28853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:29.731578 28838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:29.731750 28838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1"
format_stamp: "Formatted at 2026-09-24 01:07:29 on dist-test-slave-5c7l"
server_key: "f4a3c695f0fd411bdc01a40bd3646360"
server_key_iv: "993a271ce289afe89713f0553a719c40"
server_key_version: "encryptionkey@0"
I20260924 01:07:29.731986 28838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:29.747748 28838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:29.752187 28838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:29.752504 28838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:29.753482 28838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:29.754915 28838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:29.754992 28838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:29.755175 28838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:29.755266 28838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:29.789791 28838 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:36071
I20260924 01:07:29.789846 28965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:36071 every 8 connection(s)
I20260924 01:07:29.791260 28838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:07:29.800947 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28838
I20260924 01:07:29.801131 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:07:29.801699 9209 external_mini_cluster.cc:1428] Setting key de89ecbfdad76b31f62b8e21f94e494a
Sep 24 01:07:29 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212049, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.66@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:29.813556 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:29.794040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:42569 (local address 127.8.254.126:41789)
0924 01:07:29.794624 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:29.794632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:29.795831 (+ 1199us) server_negotiation.cc:407] Connection header received
0924 01:07:29.797770 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:29.797776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:29.797857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:29.797974 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:29.799599 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.800362 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.802758 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.803410 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.803542 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:29.807721 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:29.807727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:29.807730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:29.807766 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:29.810405 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:29.811176 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:29.811181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:29.811183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:29.811328 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:29.811782 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:29.811794 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:29.811796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:29.812052 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:29.812210 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:29.812920 (+ 710us) server_negotiation.cc:299] Negotiation successful
0924 01:07:29.813146 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":432,"thread_start_us":252,"threads_started":1}
I20260924 01:07:29.814682 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:29.794679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41789 (local address 127.8.254.66:42569)
0924 01:07:29.795660 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:29.795701 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:29.797400 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:29.798248 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:29.798260 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:29.798818 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:29.799413 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:29.799431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.800565 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.800571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:29.802588 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:29.802605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:29.803655 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:29.803665 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:29.803958 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:29.804680 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:29.804711 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:29.807467 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:29.810551 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:29.810560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:29.810579 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:29.811028 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:29.811445 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:29.811450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:29.811454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:29.811653 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:29.812184 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:29.812192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:29.812546 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 01:07:29.812832 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 01:07:29.813384 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":359,"threads_started":1}
I20260924 01:07:29.816291 28966 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41789
I20260924 01:07:29.816653 28966 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:29.817570 28966 heartbeater.cc:511] Master 127.8.254.126:41789 requested a full tablet report, sending...
I20260924 01:07:29.819197 28643 ts_manager.cc:194] Registered new tserver with Master: fc16fdb97ec044b1b0ce5086d1eef1e1 (127.8.254.66:36071)
I20260924 01:07:29.820041 28643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.66@KRBTEST.COM'} at 127.8.254.66:42569
WARNING: no policy specified for kudu/127.8.254.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
Entry for principal kudu/127.8.254.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab.
I20260924 01:07:29.859923 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41789
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:30.197511 28972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:30.197796 28972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:30.197892 28972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:30.205283 28972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:30.205389 28972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:30.205502 28972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:07:30.215426 28972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.67.keytab
--principal=kudu/127.8.254.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:41789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:30.216668 28972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:30.218168 28972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:30.231997 28977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:30.231926 28978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:30.231962 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:30.233224 28972 server_base.cc:1034] running on GCE node
I20260924 01:07:30.233958 28972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:30.235047 28972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:30.236258 28972 hybrid_clock.cc:648] HybridClock initialized: now 1790212050236209 us; error 40 us; skew 500 ppm
I20260924 01:07:30.236608 28972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:30 dist-test-slave-5c7l krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:30.239837 28972 init.cc:377] Logged in from keytab as kudu/127.8.254.67@KRBTEST.COM (short username kudu)
I20260924 01:07:30.241298 28972 webserver.cc:460] Webserver started at http://127.8.254.67:38015/ using document root <none> and password file <none>
I20260924 01:07:30.241948 28972 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:30.242048 28972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:30.242398 28972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:30.245276 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "cb04933b18724c42bd971b9fdc58c074"
format_stamp: "Formatted at 2026-09-24 01:07:30 on dist-test-slave-5c7l"
server_key: "f289c99d27111779401b99be54d0ab8d"
server_key_iv: "f731c605a1d30da9b9a0ca7d8cbdceae"
server_key_version: "encryptionkey@0"
I20260924 01:07:30.246021 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "cb04933b18724c42bd971b9fdc58c074"
format_stamp: "Formatted at 2026-09-24 01:07:30 on dist-test-slave-5c7l"
server_key: "f289c99d27111779401b99be54d0ab8d"
server_key_iv: "f731c605a1d30da9b9a0ca7d8cbdceae"
server_key_version: "encryptionkey@0"
I20260924 01:07:30.251878 28972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 01:07:30.255507 28987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:30.256878 28972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:30.257094 28972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "cb04933b18724c42bd971b9fdc58c074"
format_stamp: "Formatted at 2026-09-24 01:07:30 on dist-test-slave-5c7l"
server_key: "f289c99d27111779401b99be54d0ab8d"
server_key_iv: "f731c605a1d30da9b9a0ca7d8cbdceae"
server_key_version: "encryptionkey@0"
I20260924 01:07:30.257331 28972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:30.277658 28972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:30.281854 28972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:30.282176 28972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:30.283227 28972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:30.284564 28972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:30.284669 28972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:30.284822 28972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:30.284898 28972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:30.291678 28832 heartbeater.cc:503] Master 127.8.254.126:41789 was elected leader, sending a full tablet report...
I20260924 01:07:30.318336 28972 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:33337
I20260924 01:07:30.318361 29099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:33337 every 8 connection(s)
I20260924 01:07:30.319703 28972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:07:30.325598 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 28972
I20260924 01:07:30.325807 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:07:30.326339 9209 external_mini_cluster.cc:1428] Setting key d8a3e3b70d3b3d536a31b3947efa81a7
Sep 24 01:07:30 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.67@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:30.341696 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.322454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:36923 (local address 127.8.254.126:41789)
0924 01:07:30.322944 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.322950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.324037 (+ 1087us) server_negotiation.cc:407] Connection header received
0924 01:07:30.325976 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.325981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.326080 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.326202 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:30.328372 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.329362 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.331018 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.331424 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.331525 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.335318 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:30.335323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:30.335326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:30.335366 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:30.338286 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.339077 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.339083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.339088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.339224 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.339694 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.339698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.339700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.340077 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:30.340265 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.341014 (+ 749us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.341267 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":218,"threads_started":1}
I20260924 01:07:30.342813 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.323060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:41789 (local address 127.8.254.67:36923)
0924 01:07:30.323864 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:30.323914 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:30.325641 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:30.326522 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:30.326538 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:30.327221 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:30.328043 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.328068 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.329515 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.329520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.330878 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.330891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.331625 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.331630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.331860 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.332498 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:07:30.332526 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:30.335071 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:30.338444 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:30.338455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:30.338475 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:30.338938 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:30.339353 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:07:30.339358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:07:30.339362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:07:30.339566 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:07:30.340218 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:30.340228 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:30.340614 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 01:07:30.340910 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 01:07:30.341481 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":289,"threads_started":1}
I20260924 01:07:30.344417 29100 heartbeater.cc:347] Connected to a master server at 127.8.254.126:41789
I20260924 01:07:30.344749 29100 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:30.345505 29100 heartbeater.cc:511] Master 127.8.254.126:41789 requested a full tablet report, sending...
I20260924 01:07:30.347046 28643 ts_manager.cc:194] Registered new tserver with Master: cb04933b18724c42bd971b9fdc58c074 (127.8.254.67:33337)
I20260924 01:07:30.347899 28643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.254.67@KRBTEST.COM'} at 127.8.254.67:36923
I20260924 01:07:30.354434 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:07:30.371752 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.358785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.8.254.126:41789)
0924 01:07:30.359007 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.359014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.359435 (+ 421us) server_negotiation.cc:407] Connection header received
0924 01:07:30.359576 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.359580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.359646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.359728 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:30.361297 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.362139 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.363449 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.363861 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.363958 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.366632 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:30.366637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:30.366640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:30.366673 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:30.369251 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.369939 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.369943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.369946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.370067 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.370463 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.370467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.370469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.370792 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:30.370975 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.371248 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.371422 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:07:30.375705 28643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:07:30.382752 28643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:07:30.408440 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.395242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:36071 (local address 127.0.0.1:35956)
0924 01:07:30.397006 (+ 1764us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:30.397054 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:30.397254 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:30.398479 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:30.398488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:30.398526 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:30.398927 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.398941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.401457 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.401467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.404436 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.404458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.407082 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.407091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.407372 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.407447 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 01:07:30.407611 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 01:07:30.407895 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"thread_start_us":1347,"threads_started":1}
I20260924 01:07:30.409654 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.395445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.8.254.66:36071)
0924 01:07:30.396512 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.396523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.397494 (+ 971us) server_negotiation.cc:407] Connection header received
0924 01:07:30.397632 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.397643 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.397981 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.398177 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:30.399120 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.401239 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.404679 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.406517 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.406657 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.406903 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.408943 (+ 2040us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.409194 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":802,"threads_started":1}
I20260924 01:07:30.414156 28901 tablet_service.cc:1467] Processing CreateTablet for tablet 9c910dcd207d4567903c00af77fa2303 (DEFAULT_TABLE table=test-table [id=283c0768b520472ebc11bca2b553d2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:30.415635 28901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c910dcd207d4567903c00af77fa2303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:30.417546 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.396599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:34545 (local address 127.0.0.1:42988)
0924 01:07:30.408846 (+ 12247us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:30.408870 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:30.409011 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:30.411289 (+ 2278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:30.411294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:30.411320 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:30.411685 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.411700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.413339 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.413346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.414970 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.414986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.416770 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.416776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.416941 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.416973 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 01:07:30.417050 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 01:07:30.417184 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12078,"thread_start_us":12514,"threads_started":1}
I20260924 01:07:30.418793 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.396989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.8.254.65:34545)
0924 01:07:30.410593 (+ 13604us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.410602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.410633 (+ 31us) server_negotiation.cc:407] Connection header received
0924 01:07:30.410741 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.410750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.411072 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.411324 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:30.411885 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.413215 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.415145 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.416535 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.417960 (+ 1425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.418186 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.418305 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.418465 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13381,"thread_start_us":13177,"threads_started":1}
I20260924 01:07:30.418866 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.397592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:33337 (local address 127.0.0.1:33080)
0924 01:07:30.409743 (+ 12151us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:30.409766 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:30.409898 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:30.412254 (+ 2356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:30.412259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:30.412288 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:30.412623 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.412634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.414511 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.414517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.416080 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.416093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.418065 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.418077 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.418189 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.418280 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 01:07:30.418391 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 01:07:30.418516 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12001,"thread_start_us":11870,"threads_started":1}
I20260924 01:07:30.418936 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.398293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.8.254.67:33337)
0924 01:07:30.411668 (+ 13375us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.411676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.411709 (+ 33us) server_negotiation.cc:407] Connection header received
0924 01:07:30.411806 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.411814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.412051 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.412262 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:30.412804 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.414349 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.416232 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.417506 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.417636 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.417880 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.418447 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.418619 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12725,"thread_start_us":12265,"threads_started":1}
I20260924 01:07:30.421947 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 9c910dcd207d4567903c00af77fa2303 (DEFAULT_TABLE table=test-table [id=283c0768b520472ebc11bca2b553d2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:30.422537 28767 tablet_service.cc:1467] Processing CreateTablet for tablet 9c910dcd207d4567903c00af77fa2303 (DEFAULT_TABLE table=test-table [id=283c0768b520472ebc11bca2b553d2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:30.423345 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c910dcd207d4567903c00af77fa2303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:30.423883 28767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c910dcd207d4567903c00af77fa2303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:30.430867 29119 tablet_bootstrap.cc:492] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Bootstrap starting.
I20260924 01:07:30.434625 29119 tablet_bootstrap.cc:654] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:30.436058 29119 log.cc:824] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:30.436089 29120 tablet_bootstrap.cc:492] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Bootstrap starting.
I20260924 01:07:30.439500 29119 tablet_bootstrap.cc:492] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: No bootstrap required, opened a new log
I20260924 01:07:30.439641 29120 tablet_bootstrap.cc:654] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:30.439795 29119 ts_tablet_manager.cc:1397] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260924 01:07:30.439843 29122 tablet_bootstrap.cc:492] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Bootstrap starting.
I20260924 01:07:30.440689 29120 log.cc:824] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:30.443432 29120 tablet_bootstrap.cc:492] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: No bootstrap required, opened a new log
I20260924 01:07:30.443480 29122 tablet_bootstrap.cc:654] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:30.443761 29120 ts_tablet_manager.cc:1397] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 01:07:30.444773 29122 log.cc:824] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:30.447956 29122 tablet_bootstrap.cc:492] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: No bootstrap required, opened a new log
I20260924 01:07:30.448230 29122 ts_tablet_manager.cc:1397] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 01:07:30.450341 29119 raft_consensus.cc:348] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.450855 29119 raft_consensus.cc:374] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:30.451004 29119 raft_consensus.cc:729] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc16fdb97ec044b1b0ce5086d1eef1e1, State: Initialized, Role: FOLLOWER
I20260924 01:07:30.451718 29119 consensus_queue.cc:260] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.451828 29120 raft_consensus.cc:348] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.452219 29120 raft_consensus.cc:374] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:30.452383 29120 raft_consensus.cc:729] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb04933b18724c42bd971b9fdc58c074, State: Initialized, Role: FOLLOWER
I20260924 01:07:30.452942 28966 heartbeater.cc:503] Master 127.8.254.126:41789 was elected leader, sending a full tablet report...
I20260924 01:07:30.453156 29120 consensus_queue.cc:260] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.454274 29100 heartbeater.cc:503] Master 127.8.254.126:41789 was elected leader, sending a full tablet report...
I20260924 01:07:30.455052 29120 ts_tablet_manager.cc:1428] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 01:07:30.455054 29119 ts_tablet_manager.cc:1428] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260924 01:07:30.459859 29122 raft_consensus.cc:348] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.460374 29122 raft_consensus.cc:374] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:30.460486 29122 raft_consensus.cc:729] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d73afd424b4132b7c58d7a9bc04413, State: Initialized, Role: FOLLOWER
I20260924 01:07:30.461190 29122 consensus_queue.cc:260] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.462865 29122 ts_tablet_manager.cc:1428] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260924 01:07:30.474431 29127 raft_consensus.cc:482] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:07:30.474684 29127 raft_consensus.cc:504] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.476271 29127 leader_election.cc:288] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc16fdb97ec044b1b0ce5086d1eef1e1 (127.8.254.66:36071), cb04933b18724c42bd971b9fdc58c074 (127.8.254.67:33337)
I20260924 01:07:30.482824 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.476709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:36511 (local address 127.8.254.66:36071)
0924 01:07:30.476931 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.476937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.477517 (+ 580us) server_negotiation.cc:407] Connection header received
0924 01:07:30.477728 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.477737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.477837 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.477931 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:30.478622 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.479322 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.481055 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.481807 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.481986 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.482066 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.482434 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.482557 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:07:30.482962 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.476767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:36071 (local address 127.8.254.65:36511)
0924 01:07:30.477366 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:30.477381 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:30.477553 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:30.477962 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:30.477968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:30.478002 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:30.478469 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.478481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.479453 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.479459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.480902 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.480914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.482026 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.482032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.482253 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.482287 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:07:30.482406 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 01:07:30.482538 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":1,"thread_start_us":337,"threads_started":1}
I20260924 01:07:30.483287 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.477643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:41567 (local address 127.8.254.67:33337)
0924 01:07:30.477860 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.477869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.478289 (+ 420us) server_negotiation.cc:407] Connection header received
0924 01:07:30.478448 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.478452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.478510 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.478610 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:30.479185 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.479886 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.481534 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.482364 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.482545 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.482618 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.482908 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.483032 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 01:07:30.483335 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.477493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:33337 (local address 127.8.254.65:41567)
0924 01:07:30.478155 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:30.478171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:30.478328 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:30.478631 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:30.478636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:30.478721 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:30.479019 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.479028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.480031 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.480036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.481380 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:30.481392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.482575 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.482581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:30.482748 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.482772 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 01:07:30.482883 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 01:07:30.483010 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":294,"threads_started":1}
I20260924 01:07:30.483980 28921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c910dcd207d4567903c00af77fa2303" candidate_uuid: "d3d73afd424b4132b7c58d7a9bc04413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" is_pre_election: true
I20260924 01:07:30.484128 29055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c910dcd207d4567903c00af77fa2303" candidate_uuid: "d3d73afd424b4132b7c58d7a9bc04413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb04933b18724c42bd971b9fdc58c074" is_pre_election: true
I20260924 01:07:30.484531 28921 raft_consensus.cc:2454] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d73afd424b4132b7c58d7a9bc04413 in term 0.
I20260924 01:07:30.484647 29055 raft_consensus.cc:2454] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d73afd424b4132b7c58d7a9bc04413 in term 0.
I20260924 01:07:30.485232 28721 leader_election.cc:302] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3d73afd424b4132b7c58d7a9bc04413, fc16fdb97ec044b1b0ce5086d1eef1e1; no voters:
I20260924 01:07:30.485661 29127 raft_consensus.cc:2781] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:07:30.485822 29127 raft_consensus.cc:482] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:07:30.485973 29127 raft_consensus.cc:3037] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:30.488335 29127 raft_consensus.cc:504] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.488934 29127 leader_election.cc:288] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [CANDIDATE]: Term 1 election: Requested vote from peers fc16fdb97ec044b1b0ce5086d1eef1e1 (127.8.254.66:36071), cb04933b18724c42bd971b9fdc58c074 (127.8.254.67:33337)
I20260924 01:07:30.489387 28921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c910dcd207d4567903c00af77fa2303" candidate_uuid: "d3d73afd424b4132b7c58d7a9bc04413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1"
I20260924 01:07:30.489456 29055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c910dcd207d4567903c00af77fa2303" candidate_uuid: "d3d73afd424b4132b7c58d7a9bc04413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb04933b18724c42bd971b9fdc58c074"
I20260924 01:07:30.489614 28921 raft_consensus.cc:3037] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:30.489691 29055 raft_consensus.cc:3037] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:30.491885 28921 raft_consensus.cc:2454] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d73afd424b4132b7c58d7a9bc04413 in term 1.
I20260924 01:07:30.491987 29055 raft_consensus.cc:2454] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d73afd424b4132b7c58d7a9bc04413 in term 1.
I20260924 01:07:30.492286 28721 leader_election.cc:302] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3d73afd424b4132b7c58d7a9bc04413, fc16fdb97ec044b1b0ce5086d1eef1e1; no voters:
I20260924 01:07:30.492621 29127 raft_consensus.cc:2781] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:30.493199 29127 raft_consensus.cc:686] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 1 LEADER]: Becoming Leader. State: Replica: d3d73afd424b4132b7c58d7a9bc04413, State: Running, Role: LEADER
I20260924 01:07:30.493793 29127 consensus_queue.cc:237] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } }
I20260924 01:07:30.498602 28642 catalog_manager.cc:5495] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3d73afd424b4132b7c58d7a9bc04413 (127.8.254.65). New cstate: current_term: 1 leader_uuid: "d3d73afd424b4132b7c58d7a9bc04413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d73afd424b4132b7c58d7a9bc04413" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 34545 } health_report { overall_health: HEALTHY } } }
W20260924 01:07:30.502514 28833 tablet.cc:2369] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:30.529848 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.524898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.8.254.65:34545)
0924 01:07:30.525076 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.525080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.525098 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:07:30.525217 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.525223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.525319 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.525443 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:07:30.525780 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.526517 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.527840 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.528241 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.528333 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.528814 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:07:30.529365 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:07:30.529479 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.529568 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.529635 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 01:07:30.538689 28921 raft_consensus.cc:1261] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 1 FOLLOWER]: Refusing update from remote peer d3d73afd424b4132b7c58d7a9bc04413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:30.538748 29055 raft_consensus.cc:1261] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 1 FOLLOWER]: Refusing update from remote peer d3d73afd424b4132b7c58d7a9bc04413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:30.539667 29127 consensus_queue.cc:1035] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc16fdb97ec044b1b0ce5086d1eef1e1" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 36071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:30.540019 29130 consensus_queue.cc:1035] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb04933b18724c42bd971b9fdc58c074" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 01:07:30.543334 28967 tablet.cc:2369] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:30.553507 29134 mvcc.cc:205] Tried to move back new op lower bound from 7332708558995705856 to 7332708558826496000. Current Snapshot: MvccSnapshot[applied={T|T < 7332708558995705856 or (T in {7332708558995705856})}]
I20260924 01:07:30.569866 28643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:30.570235 28643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:30.574100 28643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d28e10f60644c2584123662a51dcca8: Sending DeleteTablet for 3 replicas of tablet 9c910dcd207d4567903c00af77fa2303
I20260924 01:07:30.575445 28767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c910dcd207d4567903c00af77fa2303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:30 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:42988
I20260924 01:07:30.575464 28901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c910dcd207d4567903c00af77fa2303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:30 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:35956
I20260924 01:07:30.575732 29035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c910dcd207d4567903c00af77fa2303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:30 UTC) from {username='kudu', principal='kudu/127.8.254.126@KRBTEST.COM'} at 127.0.0.1:33080
I20260924 01:07:30.577385 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28704
I20260924 01:07:30.586704 29145 tablet_replica.cc:330] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: stopping tablet replica
I20260924 01:07:30.587183 29145 raft_consensus.cc:2229] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:30.587718 29145 raft_consensus.cc:2258] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:30.591713 29143 tablet_replica.cc:330] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: stopping tablet replica
I20260924 01:07:30.592293 29143 raft_consensus.cc:2229] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:30.592905 29143 raft_consensus.cc:2258] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:30.593169 29145 ts_tablet_manager.cc:1905] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:30.593962 29144 tablet_replica.cc:330] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: stopping tablet replica
I20260924 01:07:30.594439 29144 raft_consensus.cc:2229] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:07:30.595394 29144 raft_consensus.cc:2258] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:30.597620 29144 ts_tablet_manager.cc:1905] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:30.600863 29145 ts_tablet_manager.cc:1918] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 24 01:07:30 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.65@KRBTEST.COM
I20260924 01:07:30.602541 29143 ts_tablet_manager.cc:1905] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:30.606673 29144 ts_tablet_manager.cc:1918] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:30.606935 29144 log.cc:1196] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/9c910dcd207d4567903c00af77fa2303
I20260924 01:07:30.607533 29144 ts_tablet_manager.cc:1939] T 9c910dcd207d4567903c00af77fa2303 P d3d73afd424b4132b7c58d7a9bc04413: Deleting consensus metadata
I20260924 01:07:30.609542 29143 ts_tablet_manager.cc:1918] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:30.609952 28627 catalog_manager.cc:4841] TS d3d73afd424b4132b7c58d7a9bc04413 (127.8.254.65:34545): tablet 9c910dcd207d4567903c00af77fa2303 (table test-table [id=283c0768b520472ebc11bca2b553d2ff]) successfully deleted
I20260924 01:07:30.610074 29145 log.cc:1196] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/9c910dcd207d4567903c00af77fa2303
I20260924 01:07:30.610549 29143 log.cc:1196] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/9c910dcd207d4567903c00af77fa2303
I20260924 01:07:30.610694 29145 ts_tablet_manager.cc:1939] T 9c910dcd207d4567903c00af77fa2303 P cb04933b18724c42bd971b9fdc58c074: Deleting consensus metadata
I20260924 01:07:30.611166 29143 ts_tablet_manager.cc:1939] T 9c910dcd207d4567903c00af77fa2303 P fc16fdb97ec044b1b0ce5086d1eef1e1: Deleting consensus metadata
I20260924 01:07:30.611606 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.579227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.8.254.65:34545)
0924 01:07:30.579452 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.579460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.590782 (+ 11322us) server_negotiation.cc:407] Connection header received
0924 01:07:30.590981 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.590994 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.591150 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.591269 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:30.593073 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.594405 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.596767 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.597476 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.597578 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.603189 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:30.603209 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:30.603229 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:30.603285 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:30.607659 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.608672 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.608681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.608685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.608917 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.609458 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.609463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.609466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.609933 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:30.610250 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.610966 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.611233 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:07:30.612649 28628 catalog_manager.cc:4841] TS cb04933b18724c42bd971b9fdc58c074 (127.8.254.67:33337): tablet 9c910dcd207d4567903c00af77fa2303 (table test-table [id=283c0768b520472ebc11bca2b553d2ff]) successfully deleted
I20260924 01:07:30.612950 28628 catalog_manager.cc:4841] TS fc16fdb97ec044b1b0ce5086d1eef1e1 (127.8.254.66:36071): tablet 9c910dcd207d4567903c00af77fa2303 (table test-table [id=283c0768b520472ebc11bca2b553d2ff]) successfully deleted
I20260924 01:07:30.613440 28827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:30.713856 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28704
I20260924 01:07:30.733932 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28838
Sep 24 01:07:30 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.66@KRBTEST.COM
I20260924 01:07:30.748785 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.734821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.8.254.66:36071)
0924 01:07:30.735031 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.735036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.735070 (+ 34us) server_negotiation.cc:407] Connection header received
0924 01:07:30.735222 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.735226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.735348 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.735466 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:30.736818 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.737538 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.738949 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.739363 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.739464 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.743581 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:30.743593 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:30.743609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:30.743638 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:30.746350 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.747027 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.747033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.747036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.747151 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.747468 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.747471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.747473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.747826 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:30.748050 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.748287 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.748453 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 01:07:30.749680 28961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:30.839407 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28838
I20260924 01:07:30.857128 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28972
Sep 24 01:07:30 dist-test-slave-5c7l krb5kdc[28598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.67@KRBTEST.COM
I20260924 01:07:30.873725 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:30.858013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.8.254.67:33337)
0924 01:07:30.858257 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:30.858264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:30.858329 (+ 65us) server_negotiation.cc:407] Connection header received
0924 01:07:30.858554 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:30.858559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:30.858624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:30.858733 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:30.860089 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.860829 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.862294 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:30.862693 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:30.862835 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:30.867549 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:30.867565 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:30.867581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:30.867622 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:30.870798 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.871574 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.871582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.871585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.871732 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:30.872157 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:30.872161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:30.872163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:30.872523 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:30.872819 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:30.873158 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 01:07:30.873393 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 01:07:30.874555 29095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:30.966352 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28972
I20260924 01:07:30.984520 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 28611
I20260924 01:07:30.985675 28673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:31.098501 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 28611
2026-09-24T01:07:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4317 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T01:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:31Z Disabled control of system clock
I20260924 01:07:31.177531 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39347
--webserver_interface=127.8.254.126
--webserver_port=0
--builtin_ntp_servers=127.8.254.84:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 01:07:31.522811 29154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:31.523103 29154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:31.523185 29154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:31.530625 29154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:31.530745 29154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:31.530782 29154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:31.530833 29154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:31.530870 29154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:31.540437 29154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:39347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:31.541864 29154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:31.543793 29154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:31.555917 29159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:31.555868 29162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:31.555960 29160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:31.556749 29154 server_base.cc:1034] running on GCE node
I20260924 01:07:31.557626 29154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:31.559170 29154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:31.560403 29154 hybrid_clock.cc:648] HybridClock initialized: now 1790212051560356 us; error 43 us; skew 500 ppm
I20260924 01:07:31.560771 29154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:31.563278 29154 webserver.cc:460] Webserver started at http://127.8.254.126:43141/ using document root <none> and password file <none>
I20260924 01:07:31.564029 29154 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:31.564167 29154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:31.564507 29154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:31.567584 29154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "09f4782b396b4fd6a3a75130f185e2c6"
format_stamp: "Formatted at 2026-09-24 01:07:31 on dist-test-slave-5c7l"
server_key: "3cfda29e8ebd353bed02c21ac661d6a7"
server_key_iv: "a63fd3fcd88bc6f1274287c0edd8b35a"
server_key_version: "encryptionkey@0"
I20260924 01:07:31.568428 29154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "09f4782b396b4fd6a3a75130f185e2c6"
format_stamp: "Formatted at 2026-09-24 01:07:31 on dist-test-slave-5c7l"
server_key: "3cfda29e8ebd353bed02c21ac661d6a7"
server_key_iv: "a63fd3fcd88bc6f1274287c0edd8b35a"
server_key_version: "encryptionkey@0"
I20260924 01:07:31.573786 29154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 01:07:31.577359 29168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:31.578810 29154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:31.579022 29154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "09f4782b396b4fd6a3a75130f185e2c6"
format_stamp: "Formatted at 2026-09-24 01:07:31 on dist-test-slave-5c7l"
server_key: "3cfda29e8ebd353bed02c21ac661d6a7"
server_key_iv: "a63fd3fcd88bc6f1274287c0edd8b35a"
server_key_version: "encryptionkey@0"
I20260924 01:07:31.579252 29154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:31.600733 29154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:31.601643 29154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:31.601961 29154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:31.620545 29154 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:39347
I20260924 01:07:31.620558 29219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:39347 every 8 connection(s)
I20260924 01:07:31.621978 29154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:31.622366 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29154
I20260924 01:07:31.622555 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:31.623246 9209 external_mini_cluster.cc:1428] Setting key 16d788b4a4971f11c728e830ec4bfc8d
I20260924 01:07:31.626782 29220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:31.639478 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:31.625267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.8.254.126:39347)
0924 01:07:31.626384 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:31.626399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:31.626450 (+ 51us) server_negotiation.cc:407] Connection header received
0924 01:07:31.628766 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:31.628795 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:31.629365 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:31.629830 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:31.630413 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:31.632513 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:31.634022 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:31.635128 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:31.635343 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:31.636450 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:31.636462 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:31.636481 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:31.637271 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:31.637369 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:31.637397 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:31.637525 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:31.637727 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 01:07:31.638085 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":334,"threads_started":1}
I20260924 01:07:31.641167 29220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6: Bootstrap starting.
I20260924 01:07:31.644420 29220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:31.645597 29220 log.cc:824] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:31.648334 29220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6: No bootstrap required, opened a new log
I20260924 01:07:31.655871 29220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f4782b396b4fd6a3a75130f185e2c6" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39347 } }
I20260924 01:07:31.656286 29220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:31.656395 29220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f4782b396b4fd6a3a75130f185e2c6, State: Initialized, Role: FOLLOWER
I20260924 01:07:31.657038 29220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f4782b396b4fd6a3a75130f185e2c6" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39347 } }
I20260924 01:07:31.657279 29220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:31.657433 29220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:31.657616 29220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:31.660259 29220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f4782b396b4fd6a3a75130f185e2c6" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39347 } }
I20260924 01:07:31.660781 29220 leader_election.cc:302] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f4782b396b4fd6a3a75130f185e2c6; no voters:
I20260924 01:07:31.661373 29220 leader_election.cc:288] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:31.661589 29225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:31.662416 29225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [term 1 LEADER]: Becoming Leader. State: Replica: 09f4782b396b4fd6a3a75130f185e2c6, State: Running, Role: LEADER
I20260924 01:07:31.662997 29225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f4782b396b4fd6a3a75130f185e2c6" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39347 } }
I20260924 01:07:31.663566 29220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:31.666710 29226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f4782b396b4fd6a3a75130f185e2c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f4782b396b4fd6a3a75130f185e2c6" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39347 } } }
I20260924 01:07:31.667106 29226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:31.667655 29227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f4782b396b4fd6a3a75130f185e2c6. Latest consensus state: current_term: 1 leader_uuid: "09f4782b396b4fd6a3a75130f185e2c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f4782b396b4fd6a3a75130f185e2c6" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 39347 } } }
I20260924 01:07:31.668033 29227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:31.670557 29232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:31.675869 29232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:31.684928 29232 catalog_manager.cc:1347] Generated new cluster ID: ad37a6628d3e40d4bfa809f216cde902
I20260924 01:07:31.685071 29232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:31.701680 29232 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:31.703042 29232 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:31.714260 29232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6: Generated new TSK 0
I20260924 01:07:31.715108 29232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:31.725401 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.65:0
--local_ip_for_outbound_sockets=127.8.254.65
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39347
--builtin_ntp_servers=127.8.254.84:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 01:07:32.091820 29244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:32.092098 29244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:32.092268 29244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:32.099573 29244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:32.099778 29244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:32.099956 29244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.65
I20260924 01:07:32.108848 29244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.254.65
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.65
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:32.110154 29244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:32.111932 29244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:32.124994 29252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:32.125020 29250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:32.124997 29249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:32.126327 29244 server_base.cc:1034] running on GCE node
I20260924 01:07:32.126981 29244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:32.128082 29244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:32.129321 29244 hybrid_clock.cc:648] HybridClock initialized: now 1790212052129277 us; error 56 us; skew 500 ppm
I20260924 01:07:32.129666 29244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:32.132140 29244 webserver.cc:460] Webserver started at http://127.8.254.65:37799/ using document root <none> and password file <none>
I20260924 01:07:32.132871 29244 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:32.133008 29244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:32.133343 29244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:32.136441 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data/instance:
uuid: "0341858908974ab2aa3780e2c17aa93b"
format_stamp: "Formatted at 2026-09-24 01:07:32 on dist-test-slave-5c7l"
server_key: "9034343437974a93c37937826e953776"
server_key_iv: "1a4d17ae660930c400ee1d7bd00f156e"
server_key_version: "encryptionkey@0"
I20260924 01:07:32.137353 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance:
uuid: "0341858908974ab2aa3780e2c17aa93b"
format_stamp: "Formatted at 2026-09-24 01:07:32 on dist-test-slave-5c7l"
server_key: "9034343437974a93c37937826e953776"
server_key_iv: "1a4d17ae660930c400ee1d7bd00f156e"
server_key_version: "encryptionkey@0"
I20260924 01:07:32.143066 29244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 01:07:32.146817 29258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:32.148231 29244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:32.148427 29244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal
uuid: "0341858908974ab2aa3780e2c17aa93b"
format_stamp: "Formatted at 2026-09-24 01:07:32 on dist-test-slave-5c7l"
server_key: "9034343437974a93c37937826e953776"
server_key_iv: "1a4d17ae660930c400ee1d7bd00f156e"
server_key_version: "encryptionkey@0"
I20260924 01:07:32.148669 29244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:32.165071 29244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:32.165963 29244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:32.166345 29244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:32.167395 29244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:32.168951 29244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:32.169076 29244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:32.169212 29244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:32.169302 29244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:32.203224 29244 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.65:46409
I20260924 01:07:32.203312 29370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.65:46409 every 8 connection(s)
I20260924 01:07:32.204576 29244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/data/info.pb
I20260924 01:07:32.205765 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29244
I20260924 01:07:32.205935 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal/instance
I20260924 01:07:32.206492 9209 external_mini_cluster.cc:1428] Setting key ba1e1e1e1dbd60b9e9531da844bf1d5c
I20260924 01:07:32.209385 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.66:0
--local_ip_for_outbound_sockets=127.8.254.66
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39347
--builtin_ntp_servers=127.8.254.84:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 01:07:32.219129 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:32.207487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.65:50669 (local address 127.8.254.126:39347)
0924 01:07:32.207968 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:32.207974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:32.209524 (+ 1550us) server_negotiation.cc:407] Connection header received
0924 01:07:32.211441 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:32.211449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:32.211538 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:32.211657 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:32.212716 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.213909 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.215541 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.216280 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.216417 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:32.218257 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:32.218263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:32.218266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:32.218286 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:32.218324 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:32.218336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:32.218435 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:32.218654 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 01:07:32.218762 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260924 01:07:32.220168 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:32.208207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39347 (local address 127.8.254.65:50669)
0924 01:07:32.209365 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:32.209418 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:32.211164 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:32.211873 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:32.211881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:32.211971 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:32.212543 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:32.212559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.214051 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.214055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:32.215378 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:32.215396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.216715 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.216726 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:32.217068 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:32.217897 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:32.217900 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:32.217940 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:32.218015 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:32.218433 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:32.218446 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:32.218456 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 01:07:32.218644 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 01:07:32.219040 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":355,"threads_started":1}
I20260924 01:07:32.221765 29371 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39347
I20260924 01:07:32.222078 29371 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:32.222867 29371 heartbeater.cc:511] Master 127.8.254.126:39347 requested a full tablet report, sending...
I20260924 01:07:32.225171 29185 ts_manager.cc:194] Registered new tserver with Master: 0341858908974ab2aa3780e2c17aa93b (127.8.254.65:46409)
I20260924 01:07:32.227461 29185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.65:50669
W20260924 01:07:32.546866 29375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:32.547153 29375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:32.547240 29375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:32.554636 29375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:32.554750 29375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:32.554860 29375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.66
I20260924 01:07:32.564236 29375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.254.66
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.66
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:32.565578 29375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:32.567423 29375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:32.580300 29383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:32.580380 29381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:32.580355 29380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:32.581274 29375 server_base.cc:1034] running on GCE node
I20260924 01:07:32.581862 29375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:32.582854 29375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:32.584128 29375 hybrid_clock.cc:648] HybridClock initialized: now 1790212052584063 us; error 55 us; skew 500 ppm
I20260924 01:07:32.584458 29375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:32.586688 29375 webserver.cc:460] Webserver started at http://127.8.254.66:40613/ using document root <none> and password file <none>
I20260924 01:07:32.587363 29375 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:32.587487 29375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:32.587806 29375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:32.590628 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data/instance:
uuid: "2b237e9237e64b40a09f2152ca85ac97"
format_stamp: "Formatted at 2026-09-24 01:07:32 on dist-test-slave-5c7l"
server_key: "38f64321fad4e38e42e82a362e2d3f58"
server_key_iv: "cdbbb4cb982c8f9d15d37b3dacbb3f77"
server_key_version: "encryptionkey@0"
I20260924 01:07:32.591588 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance:
uuid: "2b237e9237e64b40a09f2152ca85ac97"
format_stamp: "Formatted at 2026-09-24 01:07:32 on dist-test-slave-5c7l"
server_key: "38f64321fad4e38e42e82a362e2d3f58"
server_key_iv: "cdbbb4cb982c8f9d15d37b3dacbb3f77"
server_key_version: "encryptionkey@0"
I20260924 01:07:32.597344 29375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:07:32.600772 29389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:32.602281 29375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:32.602447 29375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal
uuid: "2b237e9237e64b40a09f2152ca85ac97"
format_stamp: "Formatted at 2026-09-24 01:07:32 on dist-test-slave-5c7l"
server_key: "38f64321fad4e38e42e82a362e2d3f58"
server_key_iv: "cdbbb4cb982c8f9d15d37b3dacbb3f77"
server_key_version: "encryptionkey@0"
I20260924 01:07:32.602663 29375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:32.614600 29375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:32.615379 29375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:32.615674 29375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:32.616604 29375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:32.617976 29375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:32.618095 29375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:32.618244 29375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:32.618337 29375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 01:07:32.652556 29375 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.66:41611
I20260924 01:07:32.652612 29501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.66:41611 every 8 connection(s)
I20260924 01:07:32.654069 29375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/data/info.pb
I20260924 01:07:32.658263 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29375
I20260924 01:07:32.658455 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal/instance
I20260924 01:07:32.659042 9209 external_mini_cluster.cc:1428] Setting key 12dc690bd0fec9a468c2001c04071572
I20260924 01:07:32.662506 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.254.67:0
--local_ip_for_outbound_sockets=127.8.254.67
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39347
--builtin_ntp_servers=127.8.254.84:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 01:07:32.669621 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:32.656912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:54877 (local address 127.8.254.126:39347)
0924 01:07:32.657104 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:32.657108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:32.658760 (+ 1652us) server_negotiation.cc:407] Connection header received
0924 01:07:32.661111 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:32.661116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:32.661203 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:32.661346 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:32.662933 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.663817 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.666463 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.666858 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.666958 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:32.668720 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:32.668725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:32.668728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:32.668750 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:32.668782 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:32.668794 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:32.668861 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:32.669153 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 01:07:32.669300 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 01:07:32.671123 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:32.657381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39347 (local address 127.8.254.66:54877)
0924 01:07:32.658581 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:32.658649 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:32.660715 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:32.661657 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:32.661670 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:32.661821 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:32.662739 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:32.662770 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.663979 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.663987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:32.666299 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:32.666319 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:32.667110 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:32.667118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:32.667470 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:32.668282 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:32.668288 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:32.668363 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:32.668482 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:32.668896 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:32.668911 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:32.668923 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 01:07:32.669122 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 01:07:32.669604 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":280,"threads_started":1}
I20260924 01:07:32.672708 29502 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39347
I20260924 01:07:32.673034 29502 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:32.673961 29502 heartbeater.cc:511] Master 127.8.254.126:39347 requested a full tablet report, sending...
I20260924 01:07:32.675604 29185 ts_manager.cc:194] Registered new tserver with Master: 2b237e9237e64b40a09f2152ca85ac97 (127.8.254.66:41611)
I20260924 01:07:32.676436 29185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.66:54877
W20260924 01:07:32.995914 29505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:32.996207 29505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:32.996364 29505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:33.003785 29505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:33.003894 29505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:33.004082 29505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.254.67
I20260924 01:07:33.013752 29505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.254.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.254.67
--webserver_port=0
--tserver_master_addrs=127.8.254.126:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.254.67
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:33.015128 29505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:33.016901 29505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:33.029985 29511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:33.029995 29513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:33.030105 29510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:33.030781 29505 server_base.cc:1034] running on GCE node
I20260924 01:07:33.031561 29505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:33.032675 29505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:33.033887 29505 hybrid_clock.cc:648] HybridClock initialized: now 1790212053033858 us; error 57 us; skew 500 ppm
I20260924 01:07:33.034271 29505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 01:07:33.036587 29505 webserver.cc:460] Webserver started at http://127.8.254.67:43711/ using document root <none> and password file <none>
I20260924 01:07:33.037295 29505 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:33.037408 29505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:33.037746 29505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:33.040714 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data/instance:
uuid: "5a71028cd77649bd956c3a3e2ad017a5"
format_stamp: "Formatted at 2026-09-24 01:07:33 on dist-test-slave-5c7l"
server_key: "f91a2c5da7ebd4749cb0bc551292cf6b"
server_key_iv: "531264c5f3828ce88fac6719d6753bcb"
server_key_version: "encryptionkey@0"
I20260924 01:07:33.041608 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance:
uuid: "5a71028cd77649bd956c3a3e2ad017a5"
format_stamp: "Formatted at 2026-09-24 01:07:33 on dist-test-slave-5c7l"
server_key: "f91a2c5da7ebd4749cb0bc551292cf6b"
server_key_iv: "531264c5f3828ce88fac6719d6753bcb"
server_key_version: "encryptionkey@0"
I20260924 01:07:33.046828 29505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 01:07:33.050374 29519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:33.051792 29505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:33.051964 29505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal
uuid: "5a71028cd77649bd956c3a3e2ad017a5"
format_stamp: "Formatted at 2026-09-24 01:07:33 on dist-test-slave-5c7l"
server_key: "f91a2c5da7ebd4749cb0bc551292cf6b"
server_key_iv: "531264c5f3828ce88fac6719d6753bcb"
server_key_version: "encryptionkey@0"
I20260924 01:07:33.052199 29505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:33.077772 29505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:33.078715 29505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:33.079021 29505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:33.080001 29505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 01:07:33.081516 29505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 01:07:33.081600 29505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:33.081733 29505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 01:07:33.081812 29505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:33.114816 29505 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.67:45653
I20260924 01:07:33.114909 29631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.67:45653 every 8 connection(s)
I20260924 01:07:33.116094 29505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/data/info.pb
I20260924 01:07:33.122440 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29505
I20260924 01:07:33.122615 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal/instance
I20260924 01:07:33.123188 9209 external_mini_cluster.cc:1428] Setting key d33006778dc1fe5eb69a967f38b8e541
I20260924 01:07:33.129173 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.118715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.67:53815 (local address 127.8.254.126:39347)
0924 01:07:33.118936 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.118942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.120337 (+ 1395us) server_negotiation.cc:407] Connection header received
0924 01:07:33.122070 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.122075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.122149 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.122311 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:33.123669 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.124564 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.126663 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.127089 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.127190 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.128491 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:33.128495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:33.128498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:33.128521 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:33.128550 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:33.128561 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:33.128628 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.128829 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.128933 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 01:07:33.130192 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.119288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:39347 (local address 127.8.254.67:53815)
0924 01:07:33.120179 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:33.120228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:33.121794 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:33.122545 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:33.122557 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:33.122687 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:07:33.123475 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.123500 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.124725 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.124730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.126540 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.126552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.127290 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.127295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.127486 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.128187 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 01:07:33.128190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:07:33.128229 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 01:07:33.128308 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:07:33.128648 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:07:33.128655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:07:33.128661 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 01:07:33.128804 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 01:07:33.129087 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":271,"threads_started":1}
I20260924 01:07:33.131553 29632 heartbeater.cc:347] Connected to a master server at 127.8.254.126:39347
I20260924 01:07:33.131822 29632 heartbeater.cc:464] Registering TS with master...
I20260924 01:07:33.132520 29632 heartbeater.cc:511] Master 127.8.254.126:39347 requested a full tablet report, sending...
I20260924 01:07:33.133894 29185 ts_manager.cc:194] Registered new tserver with Master: 5a71028cd77649bd956c3a3e2ad017a5 (127.8.254.67:45653)
I20260924 01:07:33.134697 29185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.254.67:53815
I20260924 01:07:33.138613 9209 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 01:07:33.148371 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.142760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.8.254.126:39347)
0924 01:07:33.142926 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.142931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.143446 (+ 515us) server_negotiation.cc:407] Connection header received
0924 01:07:33.143582 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.143585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.143648 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.143736 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:33.144173 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.144860 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.146177 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.146632 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.146726 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.147702 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:33.147706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:33.147709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:33.147726 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:33.147757 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:33.147769 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:33.147838 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.148065 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.148139 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 01:07:33.152258 29185 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 01:07:33.159672 29185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 01:07:33.199806 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.173378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.8.254.67:45653)
0924 01:07:33.189852 (+ 16474us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.189863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.189905 (+ 42us) server_negotiation.cc:407] Connection header received
0924 01:07:33.190031 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.190038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.190365 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.190579 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:33.192271 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.193708 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.197118 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.198058 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.198158 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.198492 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.199315 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.199493 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16285,"thread_start_us":16114,"threads_started":1}
I20260924 01:07:33.199942 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.176039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.8.254.66:41611)
0924 01:07:33.181314 (+ 5275us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.181329 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.191604 (+ 10275us) server_negotiation.cc:407] Connection header received
0924 01:07:33.191952 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.191961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.192204 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.192395 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:33.193245 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.194520 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.196799 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.198291 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.198438 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.198671 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.199330 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.199520 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5039,"thread_start_us":4891,"threads_started":1}
I20260924 01:07:33.200659 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.173489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.8.254.65:46409)
0924 01:07:33.190889 (+ 17400us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.190897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.190926 (+ 29us) server_negotiation.cc:407] Connection header received
0924 01:07:33.191016 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.191025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.191283 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.191489 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:33.192332 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.193544 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.197111 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.198694 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.198834 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.199094 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.200206 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.200420 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17217,"thread_start_us":17103,"threads_started":1}
I20260924 01:07:33.200048 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.173146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:46409 (local address 127.0.0.1:52686)
0924 01:07:33.190145 (+ 16999us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:33.190162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:33.190363 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:33.191492 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:33.191497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:33.191521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:33.192128 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.192142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.193712 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.193718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.196882 (+ 3164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.196897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.199222 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.199229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.199402 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.199431 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 01:07:33.199508 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 01:07:33.199711 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16877,"mutex_wait_us":309,"thread_start_us":17940,"threads_started":1}
I20260924 01:07:33.201342 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.173040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:45653 (local address 127.0.0.1:46354)
0924 01:07:33.189052 (+ 16012us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:33.189115 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:33.189366 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:33.190933 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:33.190942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:33.190975 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:33.192041 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.192058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.193912 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.193916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.196882 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.196897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.198683 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.198689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.198960 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.199078 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 01:07:33.200111 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0924 01:07:33.200309 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15633,"mutex_wait_us":341,"thread_start_us":15408,"threads_started":1}
I20260924 01:07:33.201352 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.175435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.66:41611 (local address 127.0.0.1:57794)
0924 01:07:33.191482 (+ 16047us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:33.191500 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:33.191717 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:33.192705 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:33.192709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:33.192740 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:33.193062 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.193073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.194701 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.194706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.196584 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.196597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.198840 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.198847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.199118 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.199153 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 01:07:33.200154 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0924 01:07:33.200337 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15891,"mutex_wait_us":107,"thread_start_us":14462,"threads_started":1}
I20260924 01:07:33.205282 29306 tablet_service.cc:1467] Processing CreateTablet for tablet af0a25ac970149debcac354bda5cbd0f (DEFAULT_TABLE table=test-table [id=d2df0c12af9f4b93bd899f2a9109765c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:33.205276 29437 tablet_service.cc:1467] Processing CreateTablet for tablet af0a25ac970149debcac354bda5cbd0f (DEFAULT_TABLE table=test-table [id=d2df0c12af9f4b93bd899f2a9109765c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:33.206929 29437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0a25ac970149debcac354bda5cbd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:33.207758 29306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0a25ac970149debcac354bda5cbd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:33.208417 29567 tablet_service.cc:1467] Processing CreateTablet for tablet af0a25ac970149debcac354bda5cbd0f (DEFAULT_TABLE table=test-table [id=d2df0c12af9f4b93bd899f2a9109765c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 01:07:33.209872 29567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0a25ac970149debcac354bda5cbd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:33.221747 29650 tablet_bootstrap.cc:492] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Bootstrap starting.
I20260924 01:07:33.223784 29651 tablet_bootstrap.cc:492] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Bootstrap starting.
I20260924 01:07:33.225744 29650 tablet_bootstrap.cc:654] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:33.227177 29650 log.cc:824] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:33.228483 29651 tablet_bootstrap.cc:654] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:33.229828 29651 log.cc:824] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:33.230437 29652 tablet_bootstrap.cc:492] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Bootstrap starting.
I20260924 01:07:33.230821 29371 heartbeater.cc:503] Master 127.8.254.126:39347 was elected leader, sending a full tablet report...
I20260924 01:07:33.231176 29650 tablet_bootstrap.cc:492] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: No bootstrap required, opened a new log
I20260924 01:07:33.231465 29650 ts_tablet_manager.cc:1397] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 01:07:33.234179 29652 tablet_bootstrap.cc:654] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:33.235237 29652 log.cc:824] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:33.238049 29652 tablet_bootstrap.cc:492] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: No bootstrap required, opened a new log
I20260924 01:07:33.238418 29652 ts_tablet_manager.cc:1397] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260924 01:07:33.241495 29651 tablet_bootstrap.cc:492] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: No bootstrap required, opened a new log
I20260924 01:07:33.241851 29651 ts_tablet_manager.cc:1397] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260924 01:07:33.243188 29650 raft_consensus.cc:348] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.243664 29650 raft_consensus.cc:374] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:33.243818 29650 raft_consensus.cc:729] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a71028cd77649bd956c3a3e2ad017a5, State: Initialized, Role: FOLLOWER
I20260924 01:07:33.244498 29650 consensus_queue.cc:260] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.245692 29632 heartbeater.cc:503] Master 127.8.254.126:39347 was elected leader, sending a full tablet report...
I20260924 01:07:33.246505 29650 ts_tablet_manager.cc:1428] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 01:07:33.247956 29652 raft_consensus.cc:348] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.248337 29652 raft_consensus.cc:374] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:33.248476 29652 raft_consensus.cc:729] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0341858908974ab2aa3780e2c17aa93b, State: Initialized, Role: FOLLOWER
I20260924 01:07:33.249209 29652 consensus_queue.cc:260] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.252207 29651 raft_consensus.cc:348] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.252599 29651 raft_consensus.cc:374] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:33.252714 29651 raft_consensus.cc:729] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b237e9237e64b40a09f2152ca85ac97, State: Initialized, Role: FOLLOWER
I20260924 01:07:33.253377 29651 consensus_queue.cc:260] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.254628 29502 heartbeater.cc:503] Master 127.8.254.126:39347 was elected leader, sending a full tablet report...
I20260924 01:07:33.255162 29651 ts_tablet_manager.cc:1428] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 01:07:33.256753 29652 ts_tablet_manager.cc:1428] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260924 01:07:33.305827 29658 raft_consensus.cc:482] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 01:07:33.306185 29658 raft_consensus.cc:504] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.307920 29658 leader_election.cc:288] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a71028cd77649bd956c3a3e2ad017a5 (127.8.254.67:45653), 0341858908974ab2aa3780e2c17aa93b (127.8.254.65:46409)
I20260924 01:07:33.315070 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.308368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:41813 (local address 127.8.254.67:45653)
0924 01:07:33.308747 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.308757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.309425 (+ 668us) server_negotiation.cc:407] Connection header received
0924 01:07:33.309742 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.309750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.309841 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.309945 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:33.310596 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.311310 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.313312 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.314089 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.314291 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.314361 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.314703 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.314824 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260924 01:07:33.315088 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.308467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.254.66:47533 (local address 127.8.254.65:46409)
0924 01:07:33.308659 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.308665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.309722 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 01:07:33.309918 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.309924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.310011 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.310096 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 01:07:33.310798 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.311475 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.313306 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.314061 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.314261 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.314339 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.314670 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.314813 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:07:33.315160 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.308359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.67:45653 (local address 127.8.254.66:41813)
0924 01:07:33.309298 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:33.309324 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:33.309559 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:33.309993 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:33.309997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:33.310023 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:33.310442 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.310454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.311450 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.311457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.313087 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.313100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.314311 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.314318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.314495 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.314529 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 01:07:33.314631 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 01:07:33.314782 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":76,"thread_start_us":938,"threads_started":1}
I20260924 01:07:33.315218 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.308425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.65:46409 (local address 127.8.254.66:47533)
0924 01:07:33.309625 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0924 01:07:33.309644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 01:07:33.309764 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:07:33.310194 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:07:33.310199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:07:33.310258 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 01:07:33.310631 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.310640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.311620 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.311625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.313082 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:07:33.313094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.314296 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.314304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:07:33.314484 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.314520 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 01:07:33.314628 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 01:07:33.314814 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":63,"thread_start_us":417,"threads_started":1}
I20260924 01:07:33.316241 29587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0a25ac970149debcac354bda5cbd0f" candidate_uuid: "2b237e9237e64b40a09f2152ca85ac97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a71028cd77649bd956c3a3e2ad017a5" is_pre_election: true
I20260924 01:07:33.316253 29326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0a25ac970149debcac354bda5cbd0f" candidate_uuid: "2b237e9237e64b40a09f2152ca85ac97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0341858908974ab2aa3780e2c17aa93b" is_pre_election: true
I20260924 01:07:33.316773 29587 raft_consensus.cc:2454] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b237e9237e64b40a09f2152ca85ac97 in term 0.
I20260924 01:07:33.316771 29326 raft_consensus.cc:2454] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b237e9237e64b40a09f2152ca85ac97 in term 0.
I20260924 01:07:33.317468 29393 leader_election.cc:302] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0341858908974ab2aa3780e2c17aa93b, 2b237e9237e64b40a09f2152ca85ac97, 5a71028cd77649bd956c3a3e2ad017a5; no voters:
I20260924 01:07:33.317869 29658 raft_consensus.cc:2781] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 01:07:33.318027 29658 raft_consensus.cc:482] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 01:07:33.318149 29658 raft_consensus.cc:3037] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:33.320399 29658 raft_consensus.cc:504] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.321055 29658 leader_election.cc:288] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [CANDIDATE]: Term 1 election: Requested vote from peers 5a71028cd77649bd956c3a3e2ad017a5 (127.8.254.67:45653), 0341858908974ab2aa3780e2c17aa93b (127.8.254.65:46409)
I20260924 01:07:33.321470 29587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0a25ac970149debcac354bda5cbd0f" candidate_uuid: "2b237e9237e64b40a09f2152ca85ac97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a71028cd77649bd956c3a3e2ad017a5"
I20260924 01:07:33.321561 29326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0a25ac970149debcac354bda5cbd0f" candidate_uuid: "2b237e9237e64b40a09f2152ca85ac97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0341858908974ab2aa3780e2c17aa93b"
I20260924 01:07:33.321693 29587 raft_consensus.cc:3037] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:33.321808 29326 raft_consensus.cc:3037] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:33.324083 29587 raft_consensus.cc:2454] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b237e9237e64b40a09f2152ca85ac97 in term 1.
I20260924 01:07:33.324133 29326 raft_consensus.cc:2454] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b237e9237e64b40a09f2152ca85ac97 in term 1.
I20260924 01:07:33.324558 29390 leader_election.cc:302] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b237e9237e64b40a09f2152ca85ac97, 5a71028cd77649bd956c3a3e2ad017a5; no voters:
I20260924 01:07:33.324918 29658 raft_consensus.cc:2781] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:33.325433 29658 raft_consensus.cc:686] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 1 LEADER]: Becoming Leader. State: Replica: 2b237e9237e64b40a09f2152ca85ac97, State: Running, Role: LEADER
I20260924 01:07:33.325944 29658 consensus_queue.cc:237] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } }
I20260924 01:07:33.331097 29184 catalog_manager.cc:5495] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b237e9237e64b40a09f2152ca85ac97 (127.8.254.66). New cstate: current_term: 1 leader_uuid: "2b237e9237e64b40a09f2152ca85ac97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b237e9237e64b40a09f2152ca85ac97" member_type: VOTER last_known_addr { host: "127.8.254.66" port: 41611 } health_report { overall_health: HEALTHY } } }
W20260924 01:07:33.367841 29633 tablet.cc:2369] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:07:33.377305 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.372067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.8.254.66:41611)
0924 01:07:33.372263 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.372268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.372304 (+ 36us) server_negotiation.cc:407] Connection header received
0924 01:07:33.372531 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.372535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.372627 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.372766 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:07:33.373090 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.373821 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.375266 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.375667 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.375767 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.376272 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:07:33.376809 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:07:33.376922 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.377012 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.377075 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:07:33.386015 29587 raft_consensus.cc:1261] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 1 FOLLOWER]: Refusing update from remote peer 2b237e9237e64b40a09f2152ca85ac97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:33.386351 29326 raft_consensus.cc:1261] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 1 FOLLOWER]: Refusing update from remote peer 2b237e9237e64b40a09f2152ca85ac97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:07:33.387169 29661 consensus_queue.cc:1035] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a71028cd77649bd956c3a3e2ad017a5" member_type: VOTER last_known_addr { host: "127.8.254.67" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:33.387511 29658 consensus_queue.cc:1035] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0341858908974ab2aa3780e2c17aa93b" member_type: VOTER last_known_addr { host: "127.8.254.65" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:07:33.400403 29666 mvcc.cc:205] Tried to move back new op lower bound from 7332708570657427456 to 7332708570427342848. Current Snapshot: MvccSnapshot[applied={T|T < 7332708570657427456}]
I20260924 01:07:33.402937 29664 mvcc.cc:205] Tried to move back new op lower bound from 7332708570657427456 to 7332708570427342848. Current Snapshot: MvccSnapshot[applied={T|T < 7332708570657427456 or (T in {7332708570657427456})}]
I20260924 01:07:33.414781 29184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:33.415313 29184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 01:07:33.420679 29184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f4782b396b4fd6a3a75130f185e2c6: Sending DeleteTablet for 3 replicas of tablet af0a25ac970149debcac354bda5cbd0f
I20260924 01:07:33.422133 29306 tablet_service.cc:1514] Processing DeleteTablet for tablet af0a25ac970149debcac354bda5cbd0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:33 UTC) from {username='slave'} at 127.0.0.1:52686
I20260924 01:07:33.422906 29437 tablet_service.cc:1514] Processing DeleteTablet for tablet af0a25ac970149debcac354bda5cbd0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:33 UTC) from {username='slave'} at 127.0.0.1:57794
I20260924 01:07:33.423012 29567 tablet_service.cc:1514] Processing DeleteTablet for tablet af0a25ac970149debcac354bda5cbd0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 01:07:33 UTC) from {username='slave'} at 127.0.0.1:46354
I20260924 01:07:33.423116 29674 tablet_replica.cc:330] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: stopping tablet replica
I20260924 01:07:33.423591 29674 raft_consensus.cc:2229] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:33.424180 29674 raft_consensus.cc:2258] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:33.424382 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29244
I20260924 01:07:33.426441 29674 ts_tablet_manager.cc:1905] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:33.431313 29675 tablet_replica.cc:330] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: stopping tablet replica
I20260924 01:07:33.431708 29675 raft_consensus.cc:2229] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 1 LEADER]: Raft consensus shutting down.
I20260924 01:07:33.432266 29675 raft_consensus.cc:2258] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:33.435209 29674 ts_tablet_manager.cc:1918] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:33.435480 29674 log.cc:1196] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-0/wal/wals/af0a25ac970149debcac354bda5cbd0f
I20260924 01:07:33.436218 29674 ts_tablet_manager.cc:1939] T af0a25ac970149debcac354bda5cbd0f P 0341858908974ab2aa3780e2c17aa93b: Deleting consensus metadata
I20260924 01:07:33.438513 29676 tablet_replica.cc:330] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: stopping tablet replica
I20260924 01:07:33.438571 29172 catalog_manager.cc:4841] TS 0341858908974ab2aa3780e2c17aa93b (127.8.254.65:46409): tablet af0a25ac970149debcac354bda5cbd0f (table test-table [id=d2df0c12af9f4b93bd899f2a9109765c]) successfully deleted
I20260924 01:07:33.438982 29676 raft_consensus.cc:2229] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 01:07:33.439500 29676 raft_consensus.cc:2258] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 01:07:33.441721 29676 ts_tablet_manager.cc:1905] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:33.443717 29675 ts_tablet_manager.cc:1905] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 01:07:33.448907 29676 ts_tablet_manager.cc:1918] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:33.448948 29675 ts_tablet_manager.cc:1918] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 01:07:33.449095 29676 log.cc:1196] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-2/wal/wals/af0a25ac970149debcac354bda5cbd0f
I20260924 01:07:33.449187 29675 log.cc:1196] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Deleting WAL directory at /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790211722208042-9209-0/minicluster-data/ts-1/wal/wals/af0a25ac970149debcac354bda5cbd0f
I20260924 01:07:33.449662 29676 ts_tablet_manager.cc:1939] T af0a25ac970149debcac354bda5cbd0f P 5a71028cd77649bd956c3a3e2ad017a5: Deleting consensus metadata
I20260924 01:07:33.449779 29675 ts_tablet_manager.cc:1939] T af0a25ac970149debcac354bda5cbd0f P 2b237e9237e64b40a09f2152ca85ac97: Deleting consensus metadata
I20260924 01:07:33.451735 29169 catalog_manager.cc:4841] TS 5a71028cd77649bd956c3a3e2ad017a5 (127.8.254.67:45653): tablet af0a25ac970149debcac354bda5cbd0f (table test-table [id=d2df0c12af9f4b93bd899f2a9109765c]) successfully deleted
I20260924 01:07:33.451759 29171 catalog_manager.cc:4841] TS 2b237e9237e64b40a09f2152ca85ac97 (127.8.254.66:41611): tablet af0a25ac970149debcac354bda5cbd0f (table test-table [id=d2df0c12af9f4b93bd899f2a9109765c]) successfully deleted
I20260924 01:07:33.451962 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.426364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.8.254.65:46409)
0924 01:07:33.426540 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.426547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.437775 (+ 11228us) server_negotiation.cc:407] Connection header received
0924 01:07:33.438014 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.438020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.438128 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.438272 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:33.439394 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.440415 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.441782 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.442421 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.442544 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.450001 (+ 7457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:33.450020 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:33.450042 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:33.450911 (+ 869us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:33.450983 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:33.451012 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:33.451117 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.451421 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.451562 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:07:33.453159 29366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:33.560316 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29244
I20260924 01:07:33.579821 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29375
I20260924 01:07:33.593550 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.580681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.8.254.66:41611)
0924 01:07:33.580890 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.580897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.580918 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:07:33.581048 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.581054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.581188 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.581324 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:33.581725 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.583012 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.584601 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.585290 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.585425 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.591033 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:33.591043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:33.591059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:33.592278 (+ 1219us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:33.592371 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:33.592411 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:33.592539 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.592718 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.592856 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:07:33.594511 29497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:33.727340 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29375
I20260924 01:07:33.748883 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29505
I20260924 01:07:33.755445 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:33.749657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.8.254.67:45653)
0924 01:07:33.749830 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:33.749834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:33.749920 (+ 86us) server_negotiation.cc:407] Connection header received
0924 01:07:33.750067 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:33.750071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:33.750153 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:33.750295 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:33.750631 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.751329 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.752511 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:33.752897 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:33.752987 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:33.753961 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:33.753969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:33.753985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:33.754682 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:33.754732 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:33.754753 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:33.754844 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:33.754988 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 01:07:33.755078 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 01:07:33.756110 29627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:33.845726 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29505
I20260924 01:07:33.863014 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29154
I20260924 01:07:33.864142 29215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:33.974277 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29154
2026-09-24T01:07:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2878 ms)
[----------] 6 tests from EncryptionPolicyTest (21716 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:34 dist-test-slave-5c7l krb5kdc[29684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:34 dist-test-slave-5c7l krb5kdc[29684](info): set up 2 sockets
Sep 24 01:07:34 dist-test-slave-5c7l krb5kdc[29684](info): commencing operation
krb5kdc: starting...
W20260924 01:07:35.087766 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:35 dist-test-slave-5c7l krb5kdc[29684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:35.347695 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:35.688004 29697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:35.688302 29697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:35.688473 29697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:35.695868 29697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:35.695978 29697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:35.696089 29697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:35.696197 29697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:35.696282 29697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:35.706179 29697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:35.707554 29697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:35.708920 29697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:35.720930 29703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:35.721005 29702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:35.721038 29705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:35.721984 29697 server_base.cc:1034] running on GCE node
I20260924 01:07:35.722806 29697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:35.724314 29697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:35.725529 29697 hybrid_clock.cc:648] HybridClock initialized: now 1790212055725467 us; error 57 us; skew 500 ppm
I20260924 01:07:35.725903 29697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:35 dist-test-slave-5c7l krb5kdc[29684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:35.729285 29697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:35.730895 29697 webserver.cc:460] Webserver started at http://127.0.0.1:36487/ using document root <none> and password file <none>
I20260924 01:07:35.731581 29697 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:35.731683 29697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:35.731930 29697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:35.735112 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "7f14bc75e8d740a6a87de4293e5ae091"
format_stamp: "Formatted at 2026-09-24 01:07:35 on dist-test-slave-5c7l"
server_key: "72758affc7bef32226685848bd485d8d"
server_key_iv: "dccd4b76417beec95e9d6f5d3c6b3442"
server_key_version: "encryptionkey@0"
I20260924 01:07:35.735875 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "7f14bc75e8d740a6a87de4293e5ae091"
format_stamp: "Formatted at 2026-09-24 01:07:35 on dist-test-slave-5c7l"
server_key: "72758affc7bef32226685848bd485d8d"
server_key_iv: "dccd4b76417beec95e9d6f5d3c6b3442"
server_key_version: "encryptionkey@0"
I20260924 01:07:35.741268 29697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 01:07:35.744845 29712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:35.746387 29697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:35.746574 29697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "7f14bc75e8d740a6a87de4293e5ae091"
format_stamp: "Formatted at 2026-09-24 01:07:35 on dist-test-slave-5c7l"
server_key: "72758affc7bef32226685848bd485d8d"
server_key_iv: "dccd4b76417beec95e9d6f5d3c6b3442"
server_key_version: "encryptionkey@0"
I20260924 01:07:35.746783 29697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:35.765245 29697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:35.769850 29697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:35.770169 29697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:35.789403 29697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41163
I20260924 01:07:35.789434 29763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41163 every 8 connection(s)
I20260924 01:07:35.790861 29697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:35.793857 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29697
I20260924 01:07:35.794072 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:35.794935 9209 external_mini_cluster.cc:1428] Setting key 585fa0d5ed94d9080c427262976277a7
I20260924 01:07:35.795179 29764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:35.810395 29764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091: Bootstrap starting.
Sep 24 01:07:35 dist-test-slave-5c7l krb5kdc[29684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 01:07:35.814522 29764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:35.815924 29764 log.cc:824] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:35.818851 29764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091: No bootstrap required, opened a new log
I20260924 01:07:35.820299 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:35.796945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35021 (local address 127.0.0.1:41163)
0924 01:07:35.797764 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:35.797777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:35.797822 (+ 45us) server_negotiation.cc:407] Connection header received
0924 01:07:35.799918 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:35.799944 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:35.800419 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:35.800785 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:35.802680 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:35.803987 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:35.805871 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:35.806684 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:35.806832 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:35.811799 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:35.811814 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:35.811838 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:35.811901 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:35.815954 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:35.816755 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:35.816761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:35.816767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:35.816972 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:35.817486 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:35.817491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:35.817493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:35.818020 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:35.818375 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:35.818883 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 01:07:35.819248 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":392,"threads_started":1}
I20260924 01:07:35.824618 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:35.827461 29764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f14bc75e8d740a6a87de4293e5ae091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41163 } }
I20260924 01:07:35.827977 29764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:35.828109 29764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f14bc75e8d740a6a87de4293e5ae091, State: Initialized, Role: FOLLOWER
I20260924 01:07:35.828680 29764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f14bc75e8d740a6a87de4293e5ae091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41163 } }
I20260924 01:07:35.828996 29764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:35.829164 29764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:35.829398 29764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:35.831979 29764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f14bc75e8d740a6a87de4293e5ae091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41163 } }
I20260924 01:07:35.832535 29764 leader_election.cc:302] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f14bc75e8d740a6a87de4293e5ae091; no voters:
I20260924 01:07:35.833245 29764 leader_election.cc:288] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:35.833375 29774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:35.836061 29764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:35.840498 29774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [term 1 LEADER]: Becoming Leader. State: Replica: 7f14bc75e8d740a6a87de4293e5ae091, State: Running, Role: LEADER
I20260924 01:07:35.841346 29774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f14bc75e8d740a6a87de4293e5ae091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41163 } }
I20260924 01:07:35.844911 29775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f14bc75e8d740a6a87de4293e5ae091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f14bc75e8d740a6a87de4293e5ae091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41163 } } }
I20260924 01:07:35.845396 29775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:35.846289 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:35.829531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40095 (local address 127.0.0.1:41163)
0924 01:07:35.829699 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:35.829706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:35.830188 (+ 482us) server_negotiation.cc:407] Connection header received
0924 01:07:35.830406 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:35.830411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:35.830490 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:35.830617 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:35.832123 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:35.833006 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:35.834811 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:35.835280 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:35.835450 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:35.838337 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:35.838343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:35.838348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:35.838406 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:35.841463 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:35.842417 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:35.842423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:35.842426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:35.842551 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:35.843033 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:35.843036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:35.843039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:35.843399 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:35.843773 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:35.844622 (+ 849us) server_negotiation.cc:299] Negotiation successful
0924 01:07:35.844822 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 01:07:35.849304 29782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:35.849458 29776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f14bc75e8d740a6a87de4293e5ae091. Latest consensus state: current_term: 1 leader_uuid: "7f14bc75e8d740a6a87de4293e5ae091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f14bc75e8d740a6a87de4293e5ae091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41163 } } }
I20260924 01:07:35.849869 29776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:35.856665 29782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:35.866343 29782 catalog_manager.cc:1347] Generated new cluster ID: da7c6308421d41c3ae630cf6a761b0f7
I20260924 01:07:35.866477 29782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:35.878614 29782 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:35.880525 29782 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:35.892491 29782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f14bc75e8d740a6a87de4293e5ae091: Generated new TSK 0
I20260924 01:07:35.893537 29782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:35.905843 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29697
I20260924 01:07:35.907591 29759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:36.000373 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29697
W20260924 01:07:36.018218 29682 connection.cc:535] client connection to 127.0.0.1:41163 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41163): Connection reset by peer (error 104)
2026-09-24T01:07:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:36 dist-test-slave-5c7l krb5kdc[29797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:36 dist-test-slave-5c7l krb5kdc[29797](info): set up 2 sockets
Sep 24 01:07:36 dist-test-slave-5c7l krb5kdc[29797](info): commencing operation
krb5kdc: starting...
W20260924 01:07:37.183663 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:37 dist-test-slave-5c7l krb5kdc[29797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:37.445825 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:37.790664 29810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:37.791024 29810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:37.791185 29810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:37.798576 29810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:37.798753 29810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:37.798868 29810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:37.798979 29810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:37.799058 29810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:37.809231 29810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:37.810763 29810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:37.812079 29810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:37.823931 29815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:37.823976 29816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:37.824002 29818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:37.825191 29810 server_base.cc:1034] running on GCE node
I20260924 01:07:37.826025 29810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:37.827590 29810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:37.828858 29810 hybrid_clock.cc:648] HybridClock initialized: now 1790212057828793 us; error 53 us; skew 500 ppm
I20260924 01:07:37.829218 29810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:37 dist-test-slave-5c7l krb5kdc[29797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:37.832437 29810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:37.834015 29810 webserver.cc:460] Webserver started at http://127.0.0.1:42765/ using document root <none> and password file <none>
I20260924 01:07:37.834781 29810 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:37.834932 29810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:37.835248 29810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:37.838424 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "86bda1fe56e242538670bf6af12b3dba"
format_stamp: "Formatted at 2026-09-24 01:07:37 on dist-test-slave-5c7l"
server_key: "b31b94a32ff4d8ee76214b10302c2133"
server_key_iv: "4e730cb0ec507586366708a4b2e423df"
server_key_version: "encryptionkey@0"
I20260924 01:07:37.839219 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "86bda1fe56e242538670bf6af12b3dba"
format_stamp: "Formatted at 2026-09-24 01:07:37 on dist-test-slave-5c7l"
server_key: "b31b94a32ff4d8ee76214b10302c2133"
server_key_iv: "4e730cb0ec507586366708a4b2e423df"
server_key_version: "encryptionkey@0"
I20260924 01:07:37.844730 29810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:07:37.848264 29825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:37.849582 29810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:37.849771 29810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "86bda1fe56e242538670bf6af12b3dba"
format_stamp: "Formatted at 2026-09-24 01:07:37 on dist-test-slave-5c7l"
server_key: "b31b94a32ff4d8ee76214b10302c2133"
server_key_iv: "4e730cb0ec507586366708a4b2e423df"
server_key_version: "encryptionkey@0"
I20260924 01:07:37.849992 29810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:37.869313 29810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:37.873381 29810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:37.873705 29810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:37.893126 29810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46429
I20260924 01:07:37.893132 29876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46429 every 8 connection(s)
I20260924 01:07:37.894582 29810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:37.898488 29877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:37.902391 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29810
I20260924 01:07:37.902587 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:37.903148 9209 external_mini_cluster.cc:1428] Setting key 9931be8905def2c45c0b613a1a060b19
I20260924 01:07:37.912487 29877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba: Bootstrap starting.
I20260924 01:07:37.916386 29877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:37.917752 29877 log.cc:824] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:37.918222 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:37.904937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37571 (local address 127.0.0.1:46429)
0924 01:07:37.905691 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:37.905708 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:37.905759 (+ 51us) server_negotiation.cc:407] Connection header received
0924 01:07:37.907842 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:37.907887 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:37.908438 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:37.908908 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:37.909446 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:37.911539 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:37.912931 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:37.913950 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:37.914179 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:37.915349 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:37.915360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:37.915378 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:37.916098 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:37.916196 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:37.916240 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:37.916425 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:37.916681 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 01:07:37.916890 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":301,"threads_started":1}
I20260924 01:07:37.920917 29877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba: No bootstrap required, opened a new log
I20260924 01:07:37.922010 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:37.929188 29877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bda1fe56e242538670bf6af12b3dba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260924 01:07:37.929605 29877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:37.929713 29877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bda1fe56e242538670bf6af12b3dba, State: Initialized, Role: FOLLOWER
I20260924 01:07:37.930444 29877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bda1fe56e242538670bf6af12b3dba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260924 01:07:37.930696 29877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:37.930855 29877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:37.931044 29877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:37.932821 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:37.926300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48037 (local address 127.0.0.1:46429)
0924 01:07:37.926455 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:37.926459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:37.926977 (+ 518us) server_negotiation.cc:407] Connection header received
0924 01:07:37.927202 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:37.927205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:37.927272 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:37.927387 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:37.927889 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:37.928660 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:37.930104 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:37.930626 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:37.930765 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:37.931986 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:37.931991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:37.931994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:37.932015 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:37.932052 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:37.932070 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:37.932163 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:37.932365 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 01:07:37.932409 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 01:07:37.933936 29877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bda1fe56e242538670bf6af12b3dba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260924 01:07:37.934578 29877 leader_election.cc:302] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86bda1fe56e242538670bf6af12b3dba; no voters:
I20260924 01:07:37.935128 29877 leader_election.cc:288] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:37.935304 29887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:37.936095 29887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [term 1 LEADER]: Becoming Leader. State: Replica: 86bda1fe56e242538670bf6af12b3dba, State: Running, Role: LEADER
I20260924 01:07:37.936751 29887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bda1fe56e242538670bf6af12b3dba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260924 01:07:37.937381 29877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:37.939440 29889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bda1fe56e242538670bf6af12b3dba. Latest consensus state: current_term: 1 leader_uuid: "86bda1fe56e242538670bf6af12b3dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bda1fe56e242538670bf6af12b3dba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } } }
I20260924 01:07:37.939831 29889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:37.940382 29888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86bda1fe56e242538670bf6af12b3dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bda1fe56e242538670bf6af12b3dba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } } }
I20260924 01:07:37.940650 29888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:37.943984 29893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:37.949805 29893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:37.959466 29893 catalog_manager.cc:1347] Generated new cluster ID: b1646ca36c22446995d1c33921d15f95
I20260924 01:07:37.959579 29893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:37.978353 29893 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:37.980356 29893 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:37.994361 29893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86bda1fe56e242538670bf6af12b3dba: Generated new TSK 0
I20260924 01:07:37.995270 29893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:38.007261 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29810
W20260924 01:07:38.008852 29872 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37571
W20260924 01:07:38.009719 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29810: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:38.009934 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29810
W20260924 01:07:38.027491 29795 connection.cc:535] client connection to 127.0.0.1:46429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46429): Connection reset by peer (error 104)
2026-09-24T01:07:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:38 dist-test-slave-5c7l krb5kdc[29908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:38 dist-test-slave-5c7l krb5kdc[29908](info): set up 2 sockets
Sep 24 01:07:38 dist-test-slave-5c7l krb5kdc[29908](info): commencing operation
krb5kdc: starting...
W20260924 01:07:39.123654 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:39 dist-test-slave-5c7l krb5kdc[29908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:39.386642 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:39.731058 29921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:39.731365 29921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:39.731459 29921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:39.738756 29921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:39.738879 29921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:39.738926 29921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:39.738986 29921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:39.739032 29921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:39.749205 29921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:39.750587 29921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:39.751861 29921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:39.764093 29926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:39.764045 29927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:39.764115 29929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:39.765080 29921 server_base.cc:1034] running on GCE node
I20260924 01:07:39.765878 29921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:39.767511 29921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:39.768800 29921 hybrid_clock.cc:648] HybridClock initialized: now 1790212059768736 us; error 52 us; skew 500 ppm
I20260924 01:07:39.769209 29921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:39 dist-test-slave-5c7l krb5kdc[29908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:39.777021 29921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:39.778748 29921 webserver.cc:460] Webserver started at http://127.0.0.1:45449/ using document root <none> and password file <none>
I20260924 01:07:39.779454 29921 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:39.779562 29921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:39.779829 29921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:39.782976 29921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "03fa6fa222854621a2e1c569684535f0"
format_stamp: "Formatted at 2026-09-24 01:07:39 on dist-test-slave-5c7l"
server_key: "757e4113234e31cfe186cb4b7a3fc229"
server_key_iv: "bb722f092c23da4ae938ac7d2ace94b9"
server_key_version: "encryptionkey@0"
I20260924 01:07:39.783710 29921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "03fa6fa222854621a2e1c569684535f0"
format_stamp: "Formatted at 2026-09-24 01:07:39 on dist-test-slave-5c7l"
server_key: "757e4113234e31cfe186cb4b7a3fc229"
server_key_iv: "bb722f092c23da4ae938ac7d2ace94b9"
server_key_version: "encryptionkey@0"
I20260924 01:07:39.788918 29921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:39.793136 29936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:39.794736 29921 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 01:07:39.794932 29921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "03fa6fa222854621a2e1c569684535f0"
format_stamp: "Formatted at 2026-09-24 01:07:39 on dist-test-slave-5c7l"
server_key: "757e4113234e31cfe186cb4b7a3fc229"
server_key_iv: "bb722f092c23da4ae938ac7d2ace94b9"
server_key_version: "encryptionkey@0"
I20260924 01:07:39.795128 29921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:39.801338 29921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:39.805389 29921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:39.805689 29921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:39.824812 29921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38357
I20260924 01:07:39.824856 29987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38357 every 8 connection(s)
I20260924 01:07:39.826354 29921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:39.830119 29988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:39.831773 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 29921
I20260924 01:07:39.831975 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:39.832453 9209 external_mini_cluster.cc:1428] Setting key 5f546b3909641be5cbace1615015e803
I20260924 01:07:39.840338 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:39.834312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48175 (local address 127.0.0.1:38357)
0924 01:07:39.835150 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:39.835162 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:39.835205 (+ 43us) server_negotiation.cc:407] Connection header received
0924 01:07:39.836849 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:39.836878 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:39.837233 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:39.837658 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:39.838696 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:39.838702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:39.838715 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:39.838728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:39.838793 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:39.838812 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:39.839041 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:39.839176 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 01:07:39.839380 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":370,"threads_started":1}
I20260924 01:07:39.844310 29988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0: Bootstrap starting.
I20260924 01:07:39.844405 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:39.847808 29988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:39.849135 29988 log.cc:824] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:39.851640 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:39.848982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54717 (local address 127.0.0.1:38357)
0924 01:07:39.849163 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:39.849172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:39.849639 (+ 467us) server_negotiation.cc:407] Connection header received
0924 01:07:39.849867 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:39.849872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:39.849924 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:39.850029 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:39.850989 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:39.850994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:39.850999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:39.851017 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:39.851062 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:39.851077 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:39.851197 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:39.851317 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 01:07:39.851379 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:07:39.852408 29988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0: No bootstrap required, opened a new log
I20260924 01:07:39.860565 29988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fa6fa222854621a2e1c569684535f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260924 01:07:39.860994 29988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:39.861099 29988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fa6fa222854621a2e1c569684535f0, State: Initialized, Role: FOLLOWER
I20260924 01:07:39.861769 29988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fa6fa222854621a2e1c569684535f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260924 01:07:39.862016 29988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:39.862160 29988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:39.862344 29988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:39.864877 29988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fa6fa222854621a2e1c569684535f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260924 01:07:39.865408 29988 leader_election.cc:302] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03fa6fa222854621a2e1c569684535f0; no voters:
I20260924 01:07:39.866017 29988 leader_election.cc:288] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:39.866228 29998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:39.867066 29998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [term 1 LEADER]: Becoming Leader. State: Replica: 03fa6fa222854621a2e1c569684535f0, State: Running, Role: LEADER
I20260924 01:07:39.867659 29998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fa6fa222854621a2e1c569684535f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } }
I20260924 01:07:39.868451 29988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:39.871121 30000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03fa6fa222854621a2e1c569684535f0. Latest consensus state: current_term: 1 leader_uuid: "03fa6fa222854621a2e1c569684535f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fa6fa222854621a2e1c569684535f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } } }
I20260924 01:07:39.871666 30000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:39.871157 29999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03fa6fa222854621a2e1c569684535f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fa6fa222854621a2e1c569684535f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38357 } } }
I20260924 01:07:39.876276 29999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:39.877400 30007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:39.881587 30007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:39.890252 30007 catalog_manager.cc:1347] Generated new cluster ID: 4f02c1c66c1c49a2bd4a9324d77d9b02
I20260924 01:07:39.890372 30007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:39.931924 30007 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:39.932416 30007 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:39.937901 30007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03fa6fa222854621a2e1c569684535f0: Generated new TSK 0
I20260924 01:07:39.938871 30007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:39.942934 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29921
W20260924 01:07:39.944298 29983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48175
W20260924 01:07:39.944959 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 29921: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:39.945116 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 29921
2026-09-24T01:07:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:40 dist-test-slave-5c7l krb5kdc[30019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:40 dist-test-slave-5c7l krb5kdc[30019](info): set up 2 sockets
Sep 24 01:07:40 dist-test-slave-5c7l krb5kdc[30019](info): commencing operation
krb5kdc: starting...
W20260924 01:07:41.055707 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:41 dist-test-slave-5c7l krb5kdc[30019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:41.318966 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:41.659970 30032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:41.660280 30032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:41.660462 30032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:41.668008 30032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:41.668136 30032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:41.668183 30032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:41.668283 30032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:41.668355 30032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:41.678519 30032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:41.679925 30032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:41.681442 30032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:41.693810 30037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:41.693784 30038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:41.693892 30040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:41.695048 30032 server_base.cc:1034] running on GCE node
I20260924 01:07:41.695803 30032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:41.697273 30032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:41.698478 30032 hybrid_clock.cc:648] HybridClock initialized: now 1790212061698406 us; error 61 us; skew 500 ppm
I20260924 01:07:41.698872 30032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:41 dist-test-slave-5c7l krb5kdc[30019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:41.702093 30032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:41.703697 30032 webserver.cc:460] Webserver started at http://127.0.0.1:33111/ using document root <none> and password file <none>
I20260924 01:07:41.704382 30032 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:41.704484 30032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:41.704725 30032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:41.707821 30032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "86db4c3419284199b5c8a74e0b6bf7b4"
format_stamp: "Formatted at 2026-09-24 01:07:41 on dist-test-slave-5c7l"
server_key: "f4127c0a794ea5ee42ef615ccc9f7710"
server_key_iv: "d0cb28fbbb405d0b2efb3c7083b4773a"
server_key_version: "encryptionkey@0"
I20260924 01:07:41.708572 30032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "86db4c3419284199b5c8a74e0b6bf7b4"
format_stamp: "Formatted at 2026-09-24 01:07:41 on dist-test-slave-5c7l"
server_key: "f4127c0a794ea5ee42ef615ccc9f7710"
server_key_iv: "d0cb28fbbb405d0b2efb3c7083b4773a"
server_key_version: "encryptionkey@0"
I20260924 01:07:41.713924 30032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:41.717715 30047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:41.719530 30032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:41.719696 30032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "86db4c3419284199b5c8a74e0b6bf7b4"
format_stamp: "Formatted at 2026-09-24 01:07:41 on dist-test-slave-5c7l"
server_key: "f4127c0a794ea5ee42ef615ccc9f7710"
server_key_iv: "d0cb28fbbb405d0b2efb3c7083b4773a"
server_key_version: "encryptionkey@0"
I20260924 01:07:41.719884 30032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:41.747809 30032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:41.752430 30032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:41.752677 30032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:41.771515 30032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39027
I20260924 01:07:41.771543 30098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39027 every 8 connection(s)
I20260924 01:07:41.772986 30032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:41.775156 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30032
I20260924 01:07:41.775362 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:41.775827 9209 external_mini_cluster.cc:1428] Setting key de38562053648fc468c54b76e6b55d3a
I20260924 01:07:41.777463 30099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 01:07:41 dist-test-slave-5c7l krb5kdc[30019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 01:07:41.791458 30099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4: Bootstrap starting.
I20260924 01:07:41.795713 30099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:41.797063 30099 log.cc:824] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:41.800756 30099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4: No bootstrap required, opened a new log
I20260924 01:07:41.801070 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:41.777726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51203 (local address 127.0.0.1:39027)
0924 01:07:41.778553 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:41.778567 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:41.778617 (+ 50us) server_negotiation.cc:407] Connection header received
0924 01:07:41.780696 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:41.780725 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:41.781321 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:41.781792 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:41.783291 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:41.785330 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:41.786642 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:41.787752 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:41.787983 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:41.792199 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:41.792211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:41.792245 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:41.792290 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:41.796343 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:41.797106 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:41.797114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:41.797123 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:41.797352 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:41.797825 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:41.797832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:41.797835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:41.798529 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:41.798856 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:41.799225 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 01:07:41.799697 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":326,"threads_started":1}
I20260924 01:07:41.804886 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:41.809262 30099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260924 01:07:41.809832 30099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:41.809973 30099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86db4c3419284199b5c8a74e0b6bf7b4, State: Initialized, Role: FOLLOWER
I20260924 01:07:41.810771 30099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260924 01:07:41.811070 30099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:41.811239 30099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:41.811419 30099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:41.814421 30099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260924 01:07:41.815065 30099 leader_election.cc:302] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86db4c3419284199b5c8a74e0b6bf7b4; no voters:
I20260924 01:07:41.815828 30099 leader_election.cc:288] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:41.815977 30109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:41.817916 30099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:41.818496 30109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 86db4c3419284199b5c8a74e0b6bf7b4, State: Running, Role: LEADER
I20260924 01:07:41.819239 30109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } }
I20260924 01:07:41.822762 30111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86db4c3419284199b5c8a74e0b6bf7b4. Latest consensus state: current_term: 1 leader_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } } }
I20260924 01:07:41.823155 30111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:41.824321 30110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db4c3419284199b5c8a74e0b6bf7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39027 } } }
I20260924 01:07:41.824622 30110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:41.829543 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:41.809421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42165 (local address 127.0.0.1:39027)
0924 01:07:41.809599 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:41.809605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:41.810380 (+ 775us) server_negotiation.cc:407] Connection header received
0924 01:07:41.810605 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:41.810610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:41.810690 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:41.810810 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:41.812371 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:41.813495 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:41.814887 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:41.815561 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:41.815727 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:41.819246 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:41.819254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:41.819262 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:41.819303 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:41.824067 (+ 4764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:41.825246 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:41.825256 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:41.825262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:41.825551 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:41.826163 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:41.826171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:41.826177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:41.826677 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:41.828512 (+ 1835us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:41.828700 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 01:07:41.829017 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 01:07:41.839571 30119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:41.844041 30119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:41.852548 30119 catalog_manager.cc:1347] Generated new cluster ID: 5604d881a025434cb32962a25a6fcef1
I20260924 01:07:41.852702 30119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:41.869721 30119 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:41.871158 30119 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:41.877828 30119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86db4c3419284199b5c8a74e0b6bf7b4: Generated new TSK 0
I20260924 01:07:41.878710 30119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:41.888164 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30032
I20260924 01:07:41.889343 30094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:41.981032 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30032
2026-09-24T01:07:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:42 dist-test-slave-5c7l krb5kdc[30132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:42 dist-test-slave-5c7l krb5kdc[30132](info): set up 2 sockets
Sep 24 01:07:42 dist-test-slave-5c7l krb5kdc[30132](info): commencing operation
krb5kdc: starting...
W20260924 01:07:43.099732 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:43 dist-test-slave-5c7l krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:43.362049 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:43.721632 30145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:43.721930 30145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:43.722021 30145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:43.729387 30145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:43.729506 30145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:43.729553 30145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:43.729615 30145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:43.729660 30145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:43.739742 30145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:43.741026 30145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:43.742426 30145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:43.754382 30150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:43.754453 30153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:43.754479 30151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:43.755503 30145 server_base.cc:1034] running on GCE node
I20260924 01:07:43.756392 30145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:43.757915 30145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:43.759176 30145 hybrid_clock.cc:648] HybridClock initialized: now 1790212063759099 us; error 62 us; skew 500 ppm
I20260924 01:07:43.759586 30145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:43 dist-test-slave-5c7l krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:43.762838 30145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:43.764386 30145 webserver.cc:460] Webserver started at http://127.0.0.1:42623/ using document root <none> and password file <none>
I20260924 01:07:43.765116 30145 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:43.765266 30145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:43.765606 30145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:43.768891 30145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "fedf0742f5b54cd9ac1f77e012d5a04f"
format_stamp: "Formatted at 2026-09-24 01:07:43 on dist-test-slave-5c7l"
server_key: "2e4ac12217be6d4a61bd9165a669923d"
server_key_iv: "04afe806f8e3edb9d218a7bbb37aa1b8"
server_key_version: "encryptionkey@0"
I20260924 01:07:43.769704 30145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "fedf0742f5b54cd9ac1f77e012d5a04f"
format_stamp: "Formatted at 2026-09-24 01:07:43 on dist-test-slave-5c7l"
server_key: "2e4ac12217be6d4a61bd9165a669923d"
server_key_iv: "04afe806f8e3edb9d218a7bbb37aa1b8"
server_key_version: "encryptionkey@0"
I20260924 01:07:43.775285 30145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 01:07:43.778831 30160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:43.780180 30145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:43.780385 30145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "fedf0742f5b54cd9ac1f77e012d5a04f"
format_stamp: "Formatted at 2026-09-24 01:07:43 on dist-test-slave-5c7l"
server_key: "2e4ac12217be6d4a61bd9165a669923d"
server_key_iv: "04afe806f8e3edb9d218a7bbb37aa1b8"
server_key_version: "encryptionkey@0"
I20260924 01:07:43.780597 30145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:43.812664 30145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:43.816802 30145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:43.817113 30145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:43.836156 30145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34281
I20260924 01:07:43.836180 30211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34281 every 8 connection(s)
I20260924 01:07:43.837654 30145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:43.838011 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30145
I20260924 01:07:43.838223 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:43.838914 9209 external_mini_cluster.cc:1428] Setting key 0460eb083d9447604b97bb4f8c43b817
I20260924 01:07:43.842515 30212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:43.854517 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:43.840881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42251 (local address 127.0.0.1:34281)
0924 01:07:43.841677 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:43.841692 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:43.841751 (+ 59us) server_negotiation.cc:407] Connection header received
0924 01:07:43.844091 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:43.844123 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:43.844624 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:43.845066 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:43.845626 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:43.847744 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:43.849080 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:43.850109 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:43.850328 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:43.851517 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:43.851528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:43.851549 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:43.852309 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:43.852411 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:43.852441 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:43.852556 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:43.852777 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 01:07:43.853066 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":337,"threads_started":1}
I20260924 01:07:43.856535 30212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f: Bootstrap starting.
I20260924 01:07:43.858418 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:43.859825 30212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:43.860808 30212 log.cc:824] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:43.863972 30212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f: No bootstrap required, opened a new log
I20260924 01:07:43.869521 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:43.863001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49069 (local address 127.0.0.1:34281)
0924 01:07:43.863181 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:43.863187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:43.863558 (+ 371us) server_negotiation.cc:407] Connection header received
0924 01:07:43.863704 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:43.863709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:43.863790 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:43.863901 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:43.864408 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:43.865487 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:43.866817 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:43.867522 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:43.867646 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:43.868746 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:43.868753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:43.868758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:43.868785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:43.868831 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:43.868855 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:43.868957 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:43.869069 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 01:07:43.869160 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:07:43.872928 30212 raft_consensus.cc:348] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 01:07:43.873337 30212 raft_consensus.cc:374] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:43.873456 30212 raft_consensus.cc:729] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedf0742f5b54cd9ac1f77e012d5a04f, State: Initialized, Role: FOLLOWER
I20260924 01:07:43.874044 30212 consensus_queue.cc:260] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 01:07:43.874315 30212 raft_consensus.cc:388] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:43.874459 30212 raft_consensus.cc:482] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:43.874661 30212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:43.876940 30212 raft_consensus.cc:504] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 01:07:43.877391 30212 leader_election.cc:302] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fedf0742f5b54cd9ac1f77e012d5a04f; no voters:
I20260924 01:07:43.878028 30212 leader_election.cc:288] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:43.878170 30222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:43.878994 30222 raft_consensus.cc:686] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [term 1 LEADER]: Becoming Leader. State: Replica: fedf0742f5b54cd9ac1f77e012d5a04f, State: Running, Role: LEADER
I20260924 01:07:43.879506 30222 consensus_queue.cc:237] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 01:07:43.880211 30212 sys_catalog.cc:564] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:43.882586 30224 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fedf0742f5b54cd9ac1f77e012d5a04f. Latest consensus state: current_term: 1 leader_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } } }
I20260924 01:07:43.883049 30224 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:43.883617 30223 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf0742f5b54cd9ac1f77e012d5a04f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } } }
I20260924 01:07:43.884158 30223 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:43.884874 30230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:43.890367 30230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:43.899542 30230 catalog_manager.cc:1347] Generated new cluster ID: b2d656885ca84dc38cf04179c48658d5
I20260924 01:07:43.899664 30230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:43.914932 30230 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:43.916746 30230 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:43.928946 30230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fedf0742f5b54cd9ac1f77e012d5a04f: Generated new TSK 0
I20260924 01:07:43.930020 30230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:43.939131 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30145
W20260924 01:07:43.940303 30207 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42251
W20260924 01:07:43.941001 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30145: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:43.941165 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30145
2026-09-24T01:07:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:44 dist-test-slave-5c7l krb5kdc[30243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:44 dist-test-slave-5c7l krb5kdc[30243](info): set up 2 sockets
Sep 24 01:07:44 dist-test-slave-5c7l krb5kdc[30243](info): commencing operation
krb5kdc: starting...
W20260924 01:07:45.051775 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:45 dist-test-slave-5c7l krb5kdc[30243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:45.309557 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:45.647857 30256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:45.648154 30256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:45.648262 30256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:45.656944 30256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:45.657063 30256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:45.657107 30256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:45.657173 30256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:45.657217 30256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:45.667513 30256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:45.668896 30256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:45.670300 30256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:45.682665 30261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:45.682739 30262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:45.682727 30264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:45.683375 30256 server_base.cc:1034] running on GCE node
I20260924 01:07:45.684274 30256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:45.685849 30256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:45.687088 30256 hybrid_clock.cc:648] HybridClock initialized: now 1790212065687036 us; error 47 us; skew 500 ppm
I20260924 01:07:45.687465 30256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:45 dist-test-slave-5c7l krb5kdc[30243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:45.697270 30256 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:45.699275 30256 webserver.cc:460] Webserver started at http://127.0.0.1:35851/ using document root <none> and password file <none>
I20260924 01:07:45.700019 30256 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:45.700171 30256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:45.700500 30256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:45.703686 30256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "9b04fac9e79644f58013adafe3545a11"
format_stamp: "Formatted at 2026-09-24 01:07:45 on dist-test-slave-5c7l"
server_key: "77ce7871dfb272f7af6b797a677c8478"
server_key_iv: "cb68334458327db53d02f7a1f834772d"
server_key_version: "encryptionkey@0"
I20260924 01:07:45.704478 30256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "9b04fac9e79644f58013adafe3545a11"
format_stamp: "Formatted at 2026-09-24 01:07:45 on dist-test-slave-5c7l"
server_key: "77ce7871dfb272f7af6b797a677c8478"
server_key_iv: "cb68334458327db53d02f7a1f834772d"
server_key_version: "encryptionkey@0"
I20260924 01:07:45.709832 30256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:45.713347 30271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:45.714847 30256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 01:07:45.715044 30256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "9b04fac9e79644f58013adafe3545a11"
format_stamp: "Formatted at 2026-09-24 01:07:45 on dist-test-slave-5c7l"
server_key: "77ce7871dfb272f7af6b797a677c8478"
server_key_iv: "cb68334458327db53d02f7a1f834772d"
server_key_version: "encryptionkey@0"
I20260924 01:07:45.715260 30256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:45.721120 30256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:45.725317 30256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:45.725626 30256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:45.744284 30256 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37201
I20260924 01:07:45.744330 30322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37201 every 8 connection(s)
I20260924 01:07:45.745833 30256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:45.749608 30323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:45.754428 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30256
I20260924 01:07:45.754621 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:45.755177 9209 external_mini_cluster.cc:1428] Setting key 5de4525bf59858dd854153504d56ae52
I20260924 01:07:45.761564 30323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11: Bootstrap starting.
I20260924 01:07:45.763581 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:45.757103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43235 (local address 127.0.0.1:37201)
0924 01:07:45.757902 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:45.757914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:45.757960 (+ 46us) server_negotiation.cc:407] Connection header received
0924 01:07:45.760147 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:45.760172 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:45.760524 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:45.760914 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:45.761958 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:45.761965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:45.761979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:45.761999 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:45.762059 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:45.762081 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:45.762352 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:45.762493 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 01:07:45.762670 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":356,"threads_started":1}
I20260924 01:07:45.765311 30323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:45.766651 30323 log.cc:824] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:45.767632 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:45.769613 30323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11: No bootstrap required, opened a new log
I20260924 01:07:45.774719 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:45.772037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58873 (local address 127.0.0.1:37201)
0924 01:07:45.772275 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:45.772285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:45.772703 (+ 418us) server_negotiation.cc:407] Connection header received
0924 01:07:45.772904 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:45.772910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:45.772972 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:45.773057 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:45.774016 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:45.774020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:45.774023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:45.774037 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:45.774074 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:45.774089 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:45.774177 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:45.774337 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 01:07:45.774388 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:07:45.778832 30323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b04fac9e79644f58013adafe3545a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37201 } }
I20260924 01:07:45.779244 30323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:45.779362 30323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b04fac9e79644f58013adafe3545a11, State: Initialized, Role: FOLLOWER
I20260924 01:07:45.779953 30323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b04fac9e79644f58013adafe3545a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37201 } }
I20260924 01:07:45.780211 30323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:45.780372 30323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:45.780566 30323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:45.783110 30323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b04fac9e79644f58013adafe3545a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37201 } }
I20260924 01:07:45.783631 30323 leader_election.cc:302] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b04fac9e79644f58013adafe3545a11; no voters:
I20260924 01:07:45.784194 30323 leader_election.cc:288] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:45.784345 30333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:45.785027 30333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [term 1 LEADER]: Becoming Leader. State: Replica: 9b04fac9e79644f58013adafe3545a11, State: Running, Role: LEADER
I20260924 01:07:45.785535 30333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b04fac9e79644f58013adafe3545a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37201 } }
I20260924 01:07:45.786424 30323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:45.788844 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b04fac9e79644f58013adafe3545a11. Latest consensus state: current_term: 1 leader_uuid: "9b04fac9e79644f58013adafe3545a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b04fac9e79644f58013adafe3545a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37201 } } }
I20260924 01:07:45.789367 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:45.789280 30334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b04fac9e79644f58013adafe3545a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b04fac9e79644f58013adafe3545a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37201 } } }
I20260924 01:07:45.789598 30334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:45.791229 30338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:45.796069 30338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:45.805518 30338 catalog_manager.cc:1347] Generated new cluster ID: f06672b934164dbd83ece48e28e55d2b
I20260924 01:07:45.805656 30338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:45.842983 30338 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:45.843456 30338 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:45.852415 30338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b04fac9e79644f58013adafe3545a11: Generated new TSK 0
I20260924 01:07:45.853263 30338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:45.859454 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30256
W20260924 01:07:45.860882 30318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43235
W20260924 01:07:45.861568 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30256: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:45.861707 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30256
2026-09-24T01:07:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:45 dist-test-slave-5c7l krb5kdc[30354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:45 dist-test-slave-5c7l krb5kdc[30354](info): set up 2 sockets
Sep 24 01:07:45 dist-test-slave-5c7l krb5kdc[30354](info): commencing operation
krb5kdc: starting...
W20260924 01:07:47.027782 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:47 dist-test-slave-5c7l krb5kdc[30354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:47.293418 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:47.634038 30367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:47.634357 30367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:47.634552 30367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:47.642036 30367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:47.642154 30367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:47.642284 30367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:47.642380 30367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:47.642463 30367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:47.652855 30367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:47.654340 30367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:47.655683 30367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:47.669392 30372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:47.669454 30373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:47.669678 30375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:47.670538 30367 server_base.cc:1034] running on GCE node
I20260924 01:07:47.671389 30367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:47.672917 30367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:47.674152 30367 hybrid_clock.cc:648] HybridClock initialized: now 1790212067674101 us; error 46 us; skew 500 ppm
I20260924 01:07:47.674531 30367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:47 dist-test-slave-5c7l krb5kdc[30354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:47.677698 30367 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:47.679450 30367 webserver.cc:460] Webserver started at http://127.0.0.1:36049/ using document root <none> and password file <none>
I20260924 01:07:47.680198 30367 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:47.680351 30367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:47.680673 30367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:47.683799 30367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "022d0416f9d646bc961b2c0db20d380c"
format_stamp: "Formatted at 2026-09-24 01:07:47 on dist-test-slave-5c7l"
server_key: "59d313bf50d3550b52cb7255efb0f837"
server_key_iv: "a2fc169204b0186fd0ba2e760b0f6f44"
server_key_version: "encryptionkey@0"
I20260924 01:07:47.684568 30367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "022d0416f9d646bc961b2c0db20d380c"
format_stamp: "Formatted at 2026-09-24 01:07:47 on dist-test-slave-5c7l"
server_key: "59d313bf50d3550b52cb7255efb0f837"
server_key_iv: "a2fc169204b0186fd0ba2e760b0f6f44"
server_key_version: "encryptionkey@0"
I20260924 01:07:47.690073 30367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:07:47.693588 30382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:47.695073 30367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:47.695235 30367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "022d0416f9d646bc961b2c0db20d380c"
format_stamp: "Formatted at 2026-09-24 01:07:47 on dist-test-slave-5c7l"
server_key: "59d313bf50d3550b52cb7255efb0f837"
server_key_iv: "a2fc169204b0186fd0ba2e760b0f6f44"
server_key_version: "encryptionkey@0"
I20260924 01:07:47.695433 30367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:47.709781 30367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:47.714376 30367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:47.714639 30367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:47.733426 30367 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41477
I20260924 01:07:47.733445 30433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41477 every 8 connection(s)
I20260924 01:07:47.735037 30367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:47.739020 30434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:47.739519 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30367
I20260924 01:07:47.739735 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:47.740203 9209 external_mini_cluster.cc:1428] Setting key 73f939957af97f2178e1587fc59ad21d
I20260924 01:07:47.753917 30434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c: Bootstrap starting.
Sep 24 01:07:47 dist-test-slave-5c7l krb5kdc[30354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 01:07:47.757061 30434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:47.758586 30434 log.cc:824] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:47.762144 30434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c: No bootstrap required, opened a new log
I20260924 01:07:47.766750 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:47.742185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:51075 (local address 127.0.0.1:41477)
0924 01:07:47.743003 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:47.743029 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:47.743081 (+ 52us) server_negotiation.cc:407] Connection header received
0924 01:07:47.745960 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:47.745991 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:47.746568 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:47.747065 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:47.748508 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:47.750555 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:47.751869 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:47.752921 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:47.753112 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:47.757393 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:47.757407 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:47.757434 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:47.757498 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:47.761604 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:47.762546 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:47.762555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:47.762564 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:47.762818 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:47.763288 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:47.763293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:47.763296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:47.764014 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:47.764389 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:47.764856 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 01:07:47.765251 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":317,"threads_started":1}
I20260924 01:07:47.771281 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:47.772574 30434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d0416f9d646bc961b2c0db20d380c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41477 } }
I20260924 01:07:47.772960 30434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:47.773062 30434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022d0416f9d646bc961b2c0db20d380c, State: Initialized, Role: FOLLOWER
I20260924 01:07:47.773759 30434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d0416f9d646bc961b2c0db20d380c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41477 } }
I20260924 01:07:47.774008 30434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:47.774173 30434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:47.774391 30434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:47.777453 30434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d0416f9d646bc961b2c0db20d380c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41477 } }
I20260924 01:07:47.778137 30434 leader_election.cc:302] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 022d0416f9d646bc961b2c0db20d380c; no voters:
I20260924 01:07:47.778944 30434 leader_election.cc:288] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:47.779096 30444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:47.779786 30444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [term 1 LEADER]: Becoming Leader. State: Replica: 022d0416f9d646bc961b2c0db20d380c, State: Running, Role: LEADER
I20260924 01:07:47.780514 30444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d0416f9d646bc961b2c0db20d380c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41477 } }
I20260924 01:07:47.782548 30444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022d0416f9d646bc961b2c0db20d380c. Latest consensus state: current_term: 1 leader_uuid: "022d0416f9d646bc961b2c0db20d380c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d0416f9d646bc961b2c0db20d380c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41477 } } }
I20260924 01:07:47.784294 30444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:47.783761 30446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "022d0416f9d646bc961b2c0db20d380c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d0416f9d646bc961b2c0db20d380c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41477 } } }
I20260924 01:07:47.784878 30446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:47.787611 30434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:47.788086 30448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:47.793727 30448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:47.797150 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:47.775862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:46415 (local address 127.0.0.1:41477)
0924 01:07:47.776110 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:47.776117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:47.776471 (+ 354us) server_negotiation.cc:407] Connection header received
0924 01:07:47.776641 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:47.776645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:47.776707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:47.776786 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:47.778748 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:47.780132 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:47.781988 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:47.783019 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:47.784006 (+ 987us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:47.786968 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:47.786976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:47.786981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:47.787027 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:47.791527 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:47.792943 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:47.792951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:47.792957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:47.793151 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:47.793915 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:47.793922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:47.793929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:47.794515 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:47.796097 (+ 1582us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:47.796298 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 01:07:47.796537 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:07:47.805550 30448 catalog_manager.cc:1347] Generated new cluster ID: 06c479a8f49646769da8c713ee2ef747
I20260924 01:07:47.805714 30448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:47.824692 30448 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:47.826495 30448 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:47.831890 30448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 022d0416f9d646bc961b2c0db20d380c: Generated new TSK 0
I20260924 01:07:47.832973 30448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:47.839808 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30367
I20260924 01:07:47.841183 30429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:47.934324 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30367
W20260924 01:07:47.952312 30352 connection.cc:535] client connection to 127.0.0.1:41477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41477): Connection reset by peer (error 104)
2026-09-24T01:07:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:48 dist-test-slave-5c7l krb5kdc[30466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:48 dist-test-slave-5c7l krb5kdc[30466](info): set up 2 sockets
Sep 24 01:07:48 dist-test-slave-5c7l krb5kdc[30466](info): commencing operation
krb5kdc: starting...
W20260924 01:07:49.047900 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:49 dist-test-slave-5c7l krb5kdc[30466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:49.321947 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:49.679824 30479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:49.680138 30479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:49.680325 30479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:49.688140 30479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:49.688270 30479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:49.688361 30479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:49.688472 30479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:49.688552 30479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:49.698978 30479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:49.700389 30479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:49.701756 30479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:49.714624 30485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:49.714687 30484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:49.714588 30487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:49.715616 30479 server_base.cc:1034] running on GCE node
I20260924 01:07:49.716406 30479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:49.717818 30479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:49.719012 30479 hybrid_clock.cc:648] HybridClock initialized: now 1790212069718948 us; error 52 us; skew 500 ppm
I20260924 01:07:49.719386 30479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:49 dist-test-slave-5c7l krb5kdc[30466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:49.722645 30479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:49.724166 30479 webserver.cc:460] Webserver started at http://127.0.0.1:39769/ using document root <none> and password file <none>
I20260924 01:07:49.724895 30479 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:49.725034 30479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:49.725373 30479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:49.728574 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "6d656093fa654533bc42942a5c7c86bc"
format_stamp: "Formatted at 2026-09-24 01:07:49 on dist-test-slave-5c7l"
server_key: "8e713c01361460fa1ab3b37d8d67b5e7"
server_key_iv: "f63b3b970293538c5a532d3a3ba7a0a7"
server_key_version: "encryptionkey@0"
I20260924 01:07:49.729423 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "6d656093fa654533bc42942a5c7c86bc"
format_stamp: "Formatted at 2026-09-24 01:07:49 on dist-test-slave-5c7l"
server_key: "8e713c01361460fa1ab3b37d8d67b5e7"
server_key_iv: "f63b3b970293538c5a532d3a3ba7a0a7"
server_key_version: "encryptionkey@0"
I20260924 01:07:49.734946 30479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:07:49.738662 30494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:49.740069 30479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:49.740284 30479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "6d656093fa654533bc42942a5c7c86bc"
format_stamp: "Formatted at 2026-09-24 01:07:49 on dist-test-slave-5c7l"
server_key: "8e713c01361460fa1ab3b37d8d67b5e7"
server_key_iv: "f63b3b970293538c5a532d3a3ba7a0a7"
server_key_version: "encryptionkey@0"
I20260924 01:07:49.740500 30479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:49.752959 30479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:49.757561 30479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:49.757865 30479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:49.776813 30479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46779
I20260924 01:07:49.776830 30545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46779 every 8 connection(s)
I20260924 01:07:49.778316 30479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:49.779552 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30479
I20260924 01:07:49.779754 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:49.780294 9209 external_mini_cluster.cc:1428] Setting key a45b162b1c3e4ad030999957a74d9fcd
I20260924 01:07:49.783399 30546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:49.793484 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:49.782190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:35257 (local address 127.0.0.1:46779)
0924 01:07:49.783132 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:49.783145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:49.783189 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:07:49.784858 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:49.784882 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:49.785273 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:49.785658 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:49.786301 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:49.787675 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:49.789000 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:49.789652 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:49.789856 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:49.791192 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:49.791203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:49.791218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:49.791792 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:49.791867 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:49.791896 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:49.792080 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:49.792361 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 01:07:49.792535 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":367,"threads_started":1}
I20260924 01:07:49.794512 30546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc: Bootstrap starting.
I20260924 01:07:49.798302 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:49.798673 30546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:49.799747 30546 log.cc:824] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:49.803170 30546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc: No bootstrap required, opened a new log
I20260924 01:07:49.810102 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:49.803432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:43331 (local address 127.0.0.1:46779)
0924 01:07:49.803626 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:49.803634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:49.804066 (+ 432us) server_negotiation.cc:407] Connection header received
0924 01:07:49.804315 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:49.804319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:49.804386 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:49.804497 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:49.805093 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:49.805902 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:49.807353 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:49.807838 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:49.807983 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:49.809285 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:49.809290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:49.809294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:49.809313 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:49.809353 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:49.809368 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:49.809500 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:49.809713 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 01:07:49.809775 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 01:07:49.812220 30546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d656093fa654533bc42942a5c7c86bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260924 01:07:49.812665 30546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:49.812793 30546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d656093fa654533bc42942a5c7c86bc, State: Initialized, Role: FOLLOWER
I20260924 01:07:49.813463 30546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d656093fa654533bc42942a5c7c86bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260924 01:07:49.813735 30546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:49.813906 30546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:49.814095 30546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:49.816955 30546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d656093fa654533bc42942a5c7c86bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260924 01:07:49.817492 30546 leader_election.cc:302] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d656093fa654533bc42942a5c7c86bc; no voters:
I20260924 01:07:49.818114 30546 leader_election.cc:288] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:49.818343 30556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:49.820225 30556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [term 1 LEADER]: Becoming Leader. State: Replica: 6d656093fa654533bc42942a5c7c86bc, State: Running, Role: LEADER
I20260924 01:07:49.820533 30546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:49.820891 30556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d656093fa654533bc42942a5c7c86bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260924 01:07:49.823693 30557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d656093fa654533bc42942a5c7c86bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d656093fa654533bc42942a5c7c86bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } } }
I20260924 01:07:49.824131 30557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:49.824604 30558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d656093fa654533bc42942a5c7c86bc. Latest consensus state: current_term: 1 leader_uuid: "6d656093fa654533bc42942a5c7c86bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d656093fa654533bc42942a5c7c86bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } } }
I20260924 01:07:49.824945 30558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:49.831346 30566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:49.837014 30566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:49.846380 30566 catalog_manager.cc:1347] Generated new cluster ID: 8f2ea58f32824878ac716d4be6533515
I20260924 01:07:49.846604 30566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:49.868376 30566 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:49.870026 30566 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:49.883772 30566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d656093fa654533bc42942a5c7c86bc: Generated new TSK 0
I20260924 01:07:49.884718 30566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:49.900696 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30479
W20260924 01:07:49.902016 30541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:35257
W20260924 01:07:49.902840 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30479: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:49.903002 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30479
W20260924 01:07:49.920596 30464 connection.cc:535] client connection to 127.0.0.1:46779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46779): Connection reset by peer (error 104)
2026-09-24T01:07:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:50 dist-test-slave-5c7l krb5kdc[30578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:50 dist-test-slave-5c7l krb5kdc[30578](info): set up 2 sockets
Sep 24 01:07:50 dist-test-slave-5c7l krb5kdc[30578](info): commencing operation
krb5kdc: starting...
W20260924 01:07:51.011871 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:51 dist-test-slave-5c7l krb5kdc[30578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:51.277494 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:51.616557 30591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:51.616919 30591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:51.617087 30591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:51.624406 30591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:51.624570 30591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:51.624692 30591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:51.624848 30591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:51.624933 30591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:51.635087 30591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:51.636466 30591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:51.637820 30591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:51.649951 30596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:51.650022 30599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:51.650111 30597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:51.651288 30591 server_base.cc:1034] running on GCE node
I20260924 01:07:51.652249 30591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:51.653772 30591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:51.655011 30591 hybrid_clock.cc:648] HybridClock initialized: now 1790212071654969 us; error 39 us; skew 500 ppm
I20260924 01:07:51.655391 30591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:51 dist-test-slave-5c7l krb5kdc[30578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:51.666486 30591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:51.668223 30591 webserver.cc:460] Webserver started at http://127.0.0.1:34311/ using document root <none> and password file <none>
I20260924 01:07:51.669003 30591 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:51.669160 30591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:51.669500 30591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:51.672847 30591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "3e183f174e7c4fc890acffdfdfa7ff13"
format_stamp: "Formatted at 2026-09-24 01:07:51 on dist-test-slave-5c7l"
server_key: "ee5e4b352557e1e945d33dfec1640c3b"
server_key_iv: "d610f4597e4e8b97ee860a80e9ad0a82"
server_key_version: "encryptionkey@0"
I20260924 01:07:51.673657 30591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "3e183f174e7c4fc890acffdfdfa7ff13"
format_stamp: "Formatted at 2026-09-24 01:07:51 on dist-test-slave-5c7l"
server_key: "ee5e4b352557e1e945d33dfec1640c3b"
server_key_iv: "d610f4597e4e8b97ee860a80e9ad0a82"
server_key_version: "encryptionkey@0"
I20260924 01:07:51.679834 30591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 01:07:51.684033 30606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:51.685504 30591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:07:51.685649 30591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "3e183f174e7c4fc890acffdfdfa7ff13"
format_stamp: "Formatted at 2026-09-24 01:07:51 on dist-test-slave-5c7l"
server_key: "ee5e4b352557e1e945d33dfec1640c3b"
server_key_iv: "d610f4597e4e8b97ee860a80e9ad0a82"
server_key_version: "encryptionkey@0"
I20260924 01:07:51.685825 30591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:51.691581 30591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:51.695540 30591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:51.695786 30591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:51.714344 30591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41273
I20260924 01:07:51.714388 30657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41273 every 8 connection(s)
I20260924 01:07:51.715818 30591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:51.719707 30658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:51.723651 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30591
I20260924 01:07:51.723850 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:51.724426 9209 external_mini_cluster.cc:1428] Setting key c474611f0f7dcbc36ff917d4eb4e2611
I20260924 01:07:51.732771 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:51.726402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:47445 (local address 127.0.0.1:41273)
0924 01:07:51.727431 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:51.727444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:51.727488 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:07:51.729161 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:51.729186 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:51.729565 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:51.729988 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:51.731123 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:51.731129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:51.731142 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:51.731156 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:51.731210 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:51.731230 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:51.731476 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:51.731624 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 01:07:51.731832 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":472,"threads_started":1}
I20260924 01:07:51.733016 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13: Bootstrap starting.
I20260924 01:07:51.736522 30658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:51.736557 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:51.737550 30658 log.cc:824] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:51.740686 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13: No bootstrap required, opened a new log
I20260924 01:07:51.743525 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:51.740965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:37995 (local address 127.0.0.1:41273)
0924 01:07:51.741212 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:51.741219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:51.741565 (+ 346us) server_negotiation.cc:407] Connection header received
0924 01:07:51.741761 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:51.741765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:51.741809 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:51.741881 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:51.742846 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:51.742850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:51.742854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:51.742868 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:51.742912 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:51.742927 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:51.743023 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:51.743162 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 01:07:51.743212 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:07:51.749145 30658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41273 } }
I20260924 01:07:51.749552 30658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:51.749686 30658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e183f174e7c4fc890acffdfdfa7ff13, State: Initialized, Role: FOLLOWER
I20260924 01:07:51.750360 30658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41273 } }
I20260924 01:07:51.750607 30658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:51.750771 30658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:51.750948 30658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:51.753338 30658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41273 } }
I20260924 01:07:51.753883 30658 leader_election.cc:302] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e183f174e7c4fc890acffdfdfa7ff13; no voters:
I20260924 01:07:51.754513 30658 leader_election.cc:288] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:51.754639 30668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:51.755334 30668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [term 1 LEADER]: Becoming Leader. State: Replica: 3e183f174e7c4fc890acffdfdfa7ff13, State: Running, Role: LEADER
I20260924 01:07:51.755854 30668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41273 } }
I20260924 01:07:51.756604 30658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:51.758884 30670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e183f174e7c4fc890acffdfdfa7ff13. Latest consensus state: current_term: 1 leader_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41273 } } }
I20260924 01:07:51.759253 30670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:51.759526 30669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e183f174e7c4fc890acffdfdfa7ff13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41273 } } }
I20260924 01:07:51.759986 30669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:51.762559 30674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:51.768271 30674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:51.778373 30674 catalog_manager.cc:1347] Generated new cluster ID: b5c627a9e99f483996fe66bb86cd0157
I20260924 01:07:51.778515 30674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:51.796193 30674 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:51.796746 30674 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:51.805213 30674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e183f174e7c4fc890acffdfdfa7ff13: Generated new TSK 0
I20260924 01:07:51.806075 30674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:51.812889 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30591
W20260924 01:07:51.814335 30653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:47445
W20260924 01:07:51.815063 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:51.815227 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30591
2026-09-24T01:07:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:51 dist-test-slave-5c7l krb5kdc[30689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:51 dist-test-slave-5c7l krb5kdc[30689](info): set up 2 sockets
Sep 24 01:07:51 dist-test-slave-5c7l krb5kdc[30689](info): commencing operation
krb5kdc: starting...
W20260924 01:07:52.927953 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:53 dist-test-slave-5c7l krb5kdc[30689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:53.198459 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:53.542644 30702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:53.542964 30702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:53.543164 30702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:53.550822 30702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:53.550948 30702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:53.551065 30702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:53.551172 30702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:53.551239 30702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:53.561491 30702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:53.562922 30702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:53.564340 30702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:53.576990 30710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:53.576893 30707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:53.576957 30708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:53.577661 30702 server_base.cc:1034] running on GCE node
I20260924 01:07:53.578665 30702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:53.580142 30702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:53.581368 30702 hybrid_clock.cc:648] HybridClock initialized: now 1790212073581314 us; error 37 us; skew 500 ppm
I20260924 01:07:53.581746 30702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:53 dist-test-slave-5c7l krb5kdc[30689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:53.585104 30702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:53.586624 30702 webserver.cc:460] Webserver started at http://127.0.0.1:39371/ using document root <none> and password file <none>
I20260924 01:07:53.587307 30702 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:53.587409 30702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:53.587669 30702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:53.590760 30702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e"
format_stamp: "Formatted at 2026-09-24 01:07:53 on dist-test-slave-5c7l"
server_key: "63e276cef75c24b68859063f7bb96699"
server_key_iv: "517f82bbfd2c7cda065e2a8032ee5f02"
server_key_version: "encryptionkey@0"
I20260924 01:07:53.591495 30702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e"
format_stamp: "Formatted at 2026-09-24 01:07:53 on dist-test-slave-5c7l"
server_key: "63e276cef75c24b68859063f7bb96699"
server_key_iv: "517f82bbfd2c7cda065e2a8032ee5f02"
server_key_version: "encryptionkey@0"
I20260924 01:07:53.596964 30702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:53.600632 30717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:53.602082 30702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:07:53.602502 30702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e"
format_stamp: "Formatted at 2026-09-24 01:07:53 on dist-test-slave-5c7l"
server_key: "63e276cef75c24b68859063f7bb96699"
server_key_iv: "517f82bbfd2c7cda065e2a8032ee5f02"
server_key_version: "encryptionkey@0"
I20260924 01:07:53.602716 30702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:53.623404 30702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:53.627624 30702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:53.627933 30702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:53.647151 30702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44955
I20260924 01:07:53.647159 30768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44955 every 8 connection(s)
I20260924 01:07:53.648670 30702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:53.652771 30769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:53.654699 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30702
I20260924 01:07:53.654894 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:53.655465 9209 external_mini_cluster.cc:1428] Setting key 49c85ce4dd760e9ca2732c1551934cb3
I20260924 01:07:53.666471 30769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e: Bootstrap starting.
Sep 24 01:07:53 dist-test-slave-5c7l krb5kdc[30689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 01:07:53.669864 30769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:53.671146 30769 log.cc:824] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:53.674418 30769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e: No bootstrap required, opened a new log
I20260924 01:07:53.678524 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:53.657526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:34465 (local address 127.0.0.1:44955)
0924 01:07:53.658408 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:53.658424 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:53.658468 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:07:53.660064 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:53.660088 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:53.660536 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:53.660956 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:53.662516 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:53.663857 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:53.665174 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:53.665881 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:53.666093 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:53.670472 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:53.670480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:53.670497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:53.670540 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:53.673820 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:53.674863 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:53.674871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:53.674878 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:53.675067 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:53.675608 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:53.675612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:53.675615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:53.676124 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:53.676530 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:53.677026 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 01:07:53.677375 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260924 01:07:53.682873 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:53.683954 30769 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260924 01:07:53.684360 30769 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:53.684470 30769 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5545d72e054ad2ab2eb5d70c1cbb5e, State: Initialized, Role: FOLLOWER
I20260924 01:07:53.685228 30769 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260924 01:07:53.685580 30769 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:53.685750 30769 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:53.685945 30769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:53.689452 30769 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260924 01:07:53.690121 30769 leader_election.cc:302] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5545d72e054ad2ab2eb5d70c1cbb5e; no voters:
I20260924 01:07:53.690874 30769 leader_election.cc:288] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:53.690979 30779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:53.693849 30769 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:53.694465 30779 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [term 1 LEADER]: Becoming Leader. State: Replica: be5545d72e054ad2ab2eb5d70c1cbb5e, State: Running, Role: LEADER
I20260924 01:07:53.695171 30779 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260924 01:07:53.698943 30781 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5545d72e054ad2ab2eb5d70c1cbb5e. Latest consensus state: current_term: 1 leader_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } } }
I20260924 01:07:53.699347 30780 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5545d72e054ad2ab2eb5d70c1cbb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } } }
I20260924 01:07:53.699717 30781 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:53.699717 30780 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:53.704725 30786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:53.716697 30786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:53.721069 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:53.689239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:43245 (local address 127.0.0.1:44955)
0924 01:07:53.689434 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:53.689446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:53.689811 (+ 365us) server_negotiation.cc:407] Connection header received
0924 01:07:53.689950 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:53.689955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:53.690039 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:53.690160 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:53.691997 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:53.692899 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:53.694236 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:53.694866 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:53.695009 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:53.697819 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:53.697826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:53.697831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:53.697874 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:53.711527 (+ 13653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:53.716273 (+ 4746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:53.716283 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:53.716289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:53.716569 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:53.717412 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:53.717424 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:53.717428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:53.718055 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:53.720004 (+ 1949us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:53.720260 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 01:07:53.720541 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:07:53.734045 30786 catalog_manager.cc:1347] Generated new cluster ID: f8f0e7bec77b42bc8df433e858c2586b
I20260924 01:07:53.734308 30786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:53.747303 30786 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:53.748728 30786 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:53.761967 30786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5545d72e054ad2ab2eb5d70c1cbb5e: Generated new TSK 0
I20260924 01:07:53.763147 30786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:53.771485 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30702
I20260924 01:07:53.772890 30764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:53.862957 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30702
W20260924 01:07:53.880787 30687 connection.cc:535] client connection to 127.0.0.1:44955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44955): Connection reset by peer (error 104)
2026-09-24T01:07:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2050 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:53 dist-test-slave-5c7l krb5kdc[30801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:53 dist-test-slave-5c7l krb5kdc[30801](info): set up 2 sockets
Sep 24 01:07:53 dist-test-slave-5c7l krb5kdc[30801](info): commencing operation
krb5kdc: starting...
W20260924 01:07:54.983841 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:55 dist-test-slave-5c7l krb5kdc[30801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:55.249308 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:55.588814 30814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:55.589119 30814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:55.589226 30814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:55.596724 30814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:55.596845 30814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:55.596892 30814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:55.596959 30814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:55.597003 30814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:55.607005 30814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:55.608275 30814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:55.609541 30814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:55.621767 30819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:55.621650 30822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:55.621717 30820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:55.622797 30814 server_base.cc:1034] running on GCE node
I20260924 01:07:55.623729 30814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:55.625053 30814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:55.626291 30814 hybrid_clock.cc:648] HybridClock initialized: now 1790212075626244 us; error 32 us; skew 500 ppm
I20260924 01:07:55.626596 30814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:55 dist-test-slave-5c7l krb5kdc[30801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:55.629613 30814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:55.631197 30814 webserver.cc:460] Webserver started at http://127.0.0.1:46303/ using document root <none> and password file <none>
I20260924 01:07:55.631851 30814 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:55.631953 30814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:55.632189 30814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:55.635227 30814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b"
format_stamp: "Formatted at 2026-09-24 01:07:55 on dist-test-slave-5c7l"
server_key: "03ee7fff68635fc92822c9ceabdb1bd3"
server_key_iv: "92db9800a1ad26a71d343f8631bcddfe"
server_key_version: "encryptionkey@0"
I20260924 01:07:55.635962 30814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b"
format_stamp: "Formatted at 2026-09-24 01:07:55 on dist-test-slave-5c7l"
server_key: "03ee7fff68635fc92822c9ceabdb1bd3"
server_key_iv: "92db9800a1ad26a71d343f8631bcddfe"
server_key_version: "encryptionkey@0"
I20260924 01:07:55.641379 30814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:55.644973 30829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:55.646464 30814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:55.646622 30814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b"
format_stamp: "Formatted at 2026-09-24 01:07:55 on dist-test-slave-5c7l"
server_key: "03ee7fff68635fc92822c9ceabdb1bd3"
server_key_iv: "92db9800a1ad26a71d343f8631bcddfe"
server_key_version: "encryptionkey@0"
I20260924 01:07:55.646798 30814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:55.672439 30814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:55.677047 30814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:55.677304 30814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:55.697691 30814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45071
I20260924 01:07:55.697714 30880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45071 every 8 connection(s)
I20260924 01:07:55.699293 30814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:55.703198 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:55.709408 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30814
I20260924 01:07:55.709600 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:55.710078 9209 external_mini_cluster.cc:1428] Setting key 29c455d5424975e30208e3e481f131f9
I20260924 01:07:55.714658 30881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b: Bootstrap starting.
I20260924 01:07:55.718689 30881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:55.719710 30881 log.cc:824] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:55.723050 30881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b: No bootstrap required, opened a new log
I20260924 01:07:55.725196 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:55.712036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:35821 (local address 127.0.0.1:45071)
0924 01:07:55.712811 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:55.712827 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:55.712878 (+ 51us) server_negotiation.cc:407] Connection header received
0924 01:07:55.715227 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:55.715268 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:55.715765 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:55.716176 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:55.716760 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:55.718732 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:55.720099 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:55.720987 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:55.721169 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:55.722412 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:55.722427 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:55.722447 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:55.723159 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:55.723267 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:55.723301 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:55.723472 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:55.723740 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 01:07:55.723927 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":297,"threads_started":1}
I20260924 01:07:55.729408 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:55.733566 30881 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260924 01:07:55.734089 30881 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:55.734222 30881 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbae7e8cdd4c426a9c6acd9361d79d1b, State: Initialized, Role: FOLLOWER
I20260924 01:07:55.734874 30881 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260924 01:07:55.735239 30881 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:55.735397 30881 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:55.735589 30881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:55.737972 30881 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260924 01:07:55.738579 30881 leader_election.cc:302] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbae7e8cdd4c426a9c6acd9361d79d1b; no voters:
I20260924 01:07:55.739318 30881 leader_election.cc:288] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:55.739460 30891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:55.740736 30891 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [term 1 LEADER]: Becoming Leader. State: Replica: dbae7e8cdd4c426a9c6acd9361d79d1b, State: Running, Role: LEADER
I20260924 01:07:55.741418 30891 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } }
I20260924 01:07:55.741860 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:55.734932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:46815 (local address 127.0.0.1:45071)
0924 01:07:55.735140 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:55.735145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:55.735514 (+ 369us) server_negotiation.cc:407] Connection header received
0924 01:07:55.735738 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:55.735744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:55.735820 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:55.735935 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:55.736458 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:55.737218 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:55.738641 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:55.739266 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:55.739429 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:55.740876 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:55.740882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:55.740886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:55.740907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:55.740975 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:55.741004 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:55.741075 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:55.741473 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 01:07:55.741539 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:07:55.741904 30881 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:55.744558 30892 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } } }
I20260924 01:07:55.745100 30892 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:55.744866 30893 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbae7e8cdd4c426a9c6acd9361d79d1b. Latest consensus state: current_term: 1 leader_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbae7e8cdd4c426a9c6acd9361d79d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45071 } } }
I20260924 01:07:55.745226 30893 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:55.747730 30898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:55.753418 30898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:55.762485 30898 catalog_manager.cc:1347] Generated new cluster ID: 9743a76c255e402388102805f9c35b9e
I20260924 01:07:55.762645 30898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:55.774094 30898 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:55.775977 30898 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:55.790230 30898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbae7e8cdd4c426a9c6acd9361d79d1b: Generated new TSK 0
I20260924 01:07:55.791301 30898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:55.797978 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30814
W20260924 01:07:55.799919 30876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:35821
W20260924 01:07:55.800815 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30814: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:55.800992 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30814
W20260924 01:07:55.818420 30799 connection.cc:535] client connection to 127.0.0.1:45071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45071): Connection reset by peer (error 104)
2026-09-24T01:07:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:55 dist-test-slave-5c7l krb5kdc[30912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:55 dist-test-slave-5c7l krb5kdc[30912](info): set up 2 sockets
Sep 24 01:07:55 dist-test-slave-5c7l krb5kdc[30912](info): commencing operation
krb5kdc: starting...
W20260924 01:07:56.903977 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:57 dist-test-slave-5c7l krb5kdc[30912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 01:07:57.170792 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:57.504302 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:57.504587 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:57.504772 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:57.512552 30925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:57.512687 30925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:57.512770 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:57.512856 30925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:57.512930 30925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:57.522918 30925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:57.524245 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:57.525569 30925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:57.537377 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:57.537326 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:57.537355 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:57.538290 30925 server_base.cc:1034] running on GCE node
I20260924 01:07:57.539093 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:57.540524 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:57.541772 30925 hybrid_clock.cc:648] HybridClock initialized: now 1790212077541718 us; error 43 us; skew 500 ppm
I20260924 01:07:57.542148 30925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:57 dist-test-slave-5c7l krb5kdc[30912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:57.557238 30925 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 01:07:57.558907 30925 webserver.cc:460] Webserver started at http://127.0.0.1:45517/ using document root <none> and password file <none>
I20260924 01:07:57.559646 30925 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:57.559794 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:57.560109 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:57.563292 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "2b93e4a1446d43deb4a439029d5baa5f"
format_stamp: "Formatted at 2026-09-24 01:07:57 on dist-test-slave-5c7l"
server_key: "80afbf1b3cf82784dbe2b8de7b05e0e7"
server_key_iv: "8100e6a02a4c9b5800a146f4a49ea8a4"
server_key_version: "encryptionkey@0"
I20260924 01:07:57.564077 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "2b93e4a1446d43deb4a439029d5baa5f"
format_stamp: "Formatted at 2026-09-24 01:07:57 on dist-test-slave-5c7l"
server_key: "80afbf1b3cf82784dbe2b8de7b05e0e7"
server_key_iv: "8100e6a02a4c9b5800a146f4a49ea8a4"
server_key_version: "encryptionkey@0"
I20260924 01:07:57.569411 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:07:57.572983 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:57.574364 30925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:07:57.574548 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "2b93e4a1446d43deb4a439029d5baa5f"
format_stamp: "Formatted at 2026-09-24 01:07:57 on dist-test-slave-5c7l"
server_key: "80afbf1b3cf82784dbe2b8de7b05e0e7"
server_key_iv: "8100e6a02a4c9b5800a146f4a49ea8a4"
server_key_version: "encryptionkey@0"
I20260924 01:07:57.574750 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:57.580376 30925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:57.584564 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:57.584864 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:57.603039 30925 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45707
I20260924 01:07:57.603085 30991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45707 every 8 connection(s)
I20260924 01:07:57.604451 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:57.606405 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 30925
I20260924 01:07:57.606586 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:57.607151 9209 external_mini_cluster.cc:1428] Setting key aa85953116d20daef1c892f4512fcacd
I20260924 01:07:57.608839 30992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:57.614940 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:57.609061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:56871 (local address 127.0.0.1:45707)
0924 01:07:57.609858 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:57.609870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:57.609917 (+ 47us) server_negotiation.cc:407] Connection header received
0924 01:07:57.611527 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:57.611549 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:57.611901 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:57.612306 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:57.613298 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:57.613308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:57.613323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:57.613337 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:57.613406 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:57.613432 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:57.613686 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:57.613824 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 01:07:57.614004 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":316,"threads_started":1}
I20260924 01:07:57.618480 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:57.620676 30992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f: Bootstrap starting.
I20260924 01:07:57.624127 30992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:57.625386 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:57.622785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:33303 (local address 127.0.0.1:45707)
0924 01:07:57.622974 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:57.622983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:57.623520 (+ 537us) server_negotiation.cc:407] Connection header received
0924 01:07:57.623657 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:57.623663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:57.623725 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:57.623834 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:57.624728 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:57.624734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:57.624738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:07:57.624756 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:57.624802 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:07:57.624821 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:57.624952 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:57.625014 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 01:07:57.625076 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 01:07:57.625504 30992 log.cc:824] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f: Log is configured to *not* fsync() on all Append() calls
I20260924 01:07:57.628340 30992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f: No bootstrap required, opened a new log
I20260924 01:07:57.635690 30992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93e4a1446d43deb4a439029d5baa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260924 01:07:57.636090 30992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:57.636204 30992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b93e4a1446d43deb4a439029d5baa5f, State: Initialized, Role: FOLLOWER
I20260924 01:07:57.636814 30992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93e4a1446d43deb4a439029d5baa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260924 01:07:57.637048 30992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:57.637217 30992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:57.637404 30992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:57.639823 30992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93e4a1446d43deb4a439029d5baa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260924 01:07:57.640357 30992 leader_election.cc:302] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b93e4a1446d43deb4a439029d5baa5f; no voters:
I20260924 01:07:57.641019 30992 leader_election.cc:288] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:57.641206 31002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:57.642413 31002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [term 1 LEADER]: Becoming Leader. State: Replica: 2b93e4a1446d43deb4a439029d5baa5f, State: Running, Role: LEADER
I20260924 01:07:57.642964 31002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93e4a1446d43deb4a439029d5baa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260924 01:07:57.643388 30992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:57.645825 31003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b93e4a1446d43deb4a439029d5baa5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93e4a1446d43deb4a439029d5baa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } } }
I20260924 01:07:57.646278 31003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:57.645931 31004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b93e4a1446d43deb4a439029d5baa5f. Latest consensus state: current_term: 1 leader_uuid: "2b93e4a1446d43deb4a439029d5baa5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93e4a1446d43deb4a439029d5baa5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } } }
I20260924 01:07:57.646705 31004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:57.648378 31008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:57.653332 31008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:57.662930 31008 catalog_manager.cc:1347] Generated new cluster ID: 6be4cdbdc2ae4b7dadca0331f604259e
I20260924 01:07:57.663045 31008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:57.697453 31008 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:57.697911 31008 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:57.714748 31008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b93e4a1446d43deb4a439029d5baa5f: Generated new TSK 0
I20260924 01:07:57.715651 31008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:57.729341 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30925
W20260924 01:07:57.730557 30987 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.50:56871
W20260924 01:07:57.731273 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 30925: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:07:57.731431 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 30925
2026-09-24T01:07:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:57 dist-test-slave-5c7l krb5kdc[31023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:57 dist-test-slave-5c7l krb5kdc[31023](info): set up 2 sockets
Sep 24 01:07:57 dist-test-slave-5c7l krb5kdc[31023](info): commencing operation
krb5kdc: starting...
W20260924 01:07:58.839757 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:07:59 dist-test-slave-5c7l krb5kdc[31023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:07:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:07:59.105197 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:42605
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:42605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:07:59.439828 31036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:07:59.440147 31036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:07:59.440315 31036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:07:59.447654 31036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:07:59.447782 31036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:07:59.447892 31036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:07:59.447994 31036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:07:59.448073 31036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:07:59.458289 31036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:42605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:42605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:07:59.459656 31036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:07:59.461032 31036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:07:59.473104 31041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:59.473217 31042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:07:59.473171 31044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:07:59.474409 31036 server_base.cc:1034] running on GCE node
I20260924 01:07:59.475270 31036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:07:59.476809 31036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:07:59.478013 31036 hybrid_clock.cc:648] HybridClock initialized: now 1790212079477964 us; error 39 us; skew 500 ppm
I20260924 01:07:59.478444 31036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:07:59 dist-test-slave-5c7l krb5kdc[31023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212079, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:07:59.481613 31036 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:07:59.483116 31036 webserver.cc:460] Webserver started at http://127.8.254.126:46841/ using document root <none> and password file <none>
I20260924 01:07:59.483757 31036 fs_manager.cc:359] Metadata directory not provided
I20260924 01:07:59.483856 31036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:07:59.484100 31036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:07:59.487445 31036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "585330cff7084070bd741c3be65f3607"
format_stamp: "Formatted at 2026-09-24 01:07:59 on dist-test-slave-5c7l"
server_key: "f02b563160da547109e9afabce347e65"
server_key_iv: "26988d3443d9ed62fe30051c8b51efbe"
server_key_version: "encryptionkey@0"
I20260924 01:07:59.488620 31036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "585330cff7084070bd741c3be65f3607"
format_stamp: "Formatted at 2026-09-24 01:07:59 on dist-test-slave-5c7l"
server_key: "f02b563160da547109e9afabce347e65"
server_key_iv: "26988d3443d9ed62fe30051c8b51efbe"
server_key_version: "encryptionkey@0"
I20260924 01:07:59.495918 31036 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.007s
I20260924 01:07:59.499821 31051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:07:59.501178 31036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:07:59.501379 31036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "585330cff7084070bd741c3be65f3607"
format_stamp: "Formatted at 2026-09-24 01:07:59 on dist-test-slave-5c7l"
server_key: "f02b563160da547109e9afabce347e65"
server_key_iv: "26988d3443d9ed62fe30051c8b51efbe"
server_key_version: "encryptionkey@0"
I20260924 01:07:59.501585 31036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:07:59.537967 31036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:07:59.542387 31036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:07:59.542701 31036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:07:59.561918 31036 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:42605
I20260924 01:07:59.561937 31102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:42605 every 8 connection(s)
I20260924 01:07:59.563447 31036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:07:59.567363 31103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:07:59.571357 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31036
I20260924 01:07:59.571538 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:07:59.572077 9209 external_mini_cluster.cc:1428] Setting key da017c1b4af07e5b23c38581e41e544f
I20260924 01:07:59.580875 31103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607: Bootstrap starting.
I20260924 01:07:59.584846 31103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607: Neither blocks nor log segments found. Creating new log.
I20260924 01:07:59.586282 31103 log.cc:824] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607: Log is configured to *not* fsync() on all Append() calls
Sep 24 01:07:59 dist-test-slave-5c7l krb5kdc[31023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:07:59.589425 31103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607: No bootstrap required, opened a new log
I20260924 01:07:59.596109 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:59.573969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60307 (local address 127.8.254.126:42605)
0924 01:07:59.574833 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:59.574844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:59.574899 (+ 55us) server_negotiation.cc:407] Connection header received
0924 01:07:59.576681 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:59.576706 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:59.577143 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:59.577538 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:59.579283 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:59.580606 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:59.582390 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:59.583079 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:59.583210 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:59.588524 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:59.588532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:59.588547 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:59.588590 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:59.591713 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:59.592631 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:59.592637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:59.592643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:59.592833 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:59.593343 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:59.593352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:59.593354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:59.593884 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:59.594132 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:59.594689 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 01:07:59.595002 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":429,"threads_started":1}
I20260924 01:07:59.598866 31103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585330cff7084070bd741c3be65f3607" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42605 } }
I20260924 01:07:59.599295 31103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:07:59.599412 31103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585330cff7084070bd741c3be65f3607, State: Initialized, Role: FOLLOWER
I20260924 01:07:59.600045 31103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585330cff7084070bd741c3be65f3607" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42605 } }
I20260924 01:07:59.600351 31103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:07:59.600386 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:07:59.600510 31103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:07:59.600677 31103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:07:59.602964 31103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585330cff7084070bd741c3be65f3607" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42605 } }
I20260924 01:07:59.603575 31103 leader_election.cc:302] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 585330cff7084070bd741c3be65f3607; no voters:
I20260924 01:07:59.604261 31103 leader_election.cc:288] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:07:59.604355 31112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:07:59.606078 31112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [term 1 LEADER]: Becoming Leader. State: Replica: 585330cff7084070bd741c3be65f3607, State: Running, Role: LEADER
I20260924 01:07:59.606950 31112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585330cff7084070bd741c3be65f3607" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42605 } }
I20260924 01:07:59.609400 31112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 585330cff7084070bd741c3be65f3607. Latest consensus state: current_term: 1 leader_uuid: "585330cff7084070bd741c3be65f3607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585330cff7084070bd741c3be65f3607" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42605 } } }
I20260924 01:07:59.611400 31112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:59.610893 31114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "585330cff7084070bd741c3be65f3607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585330cff7084070bd741c3be65f3607" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 42605 } } }
I20260924 01:07:59.611946 31114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [sys.catalog]: This master's current role is: LEADER
I20260924 01:07:59.616097 31103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:07:59.616305 31117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:07:59.626019 31117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:07:59.626804 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:07:59.605604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55619 (local address 127.8.254.126:42605)
0924 01:07:59.605755 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 01:07:59.605761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:07:59.606079 (+ 318us) server_negotiation.cc:407] Connection header received
0924 01:07:59.606388 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:07:59.606394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:07:59.606471 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:07:59.606541 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:07:59.608386 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:59.609459 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:59.611423 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:07:59.612339 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:07:59.612504 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:07:59.616136 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:07:59.616144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:07:59.616149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:07:59.616197 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:07:59.620961 (+ 4764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:59.622304 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:59.622316 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:59.622322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:59.622517 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:07:59.623323 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:07:59.623331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:07:59.623336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:07:59.624021 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:07:59.625678 (+ 1657us) server_negotiation.cc:1035] Waiting for connection context
0924 01:07:59.625943 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 01:07:59.626232 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 01:07:59.647321 31117 catalog_manager.cc:1347] Generated new cluster ID: df01f2463bc342a6aeebbe71f1b734db
I20260924 01:07:59.647462 31117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:07:59.678901 31117 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:07:59.680270 31117 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:07:59.702220 31117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 585330cff7084070bd741c3be65f3607: Generated new TSK 0
I20260924 01:07:59.703159 31117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:07:59.714632 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31036
I20260924 01:07:59.716197 31098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:07:59.811920 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31036
W20260924 01:07:59.829627 31021 connection.cc:535] client connection to 127.8.254.126:42605 recv error: Network error: failed to read from TLS socket (remote: 127.8.254.126:42605): Connection reset by peer (error 104)
2026-09-24T01:07:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:07:59 dist-test-slave-5c7l krb5kdc[31135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:07:59 dist-test-slave-5c7l krb5kdc[31135](info): set up 2 sockets
Sep 24 01:07:59 dist-test-slave-5c7l krb5kdc[31135](info): commencing operation
krb5kdc: starting...
W20260924 01:08:00.931856 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:01 dist-test-slave-5c7l krb5kdc[31135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:01.198019 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:46587
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:46587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:01.534889 31148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:01.535265 31148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:01.535434 31148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:01.542865 31148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:01.543040 31148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:01.543150 31148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:01.543267 31148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:01.543351 31148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:01.553387 31148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:46587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:46587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:01.554862 31148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:01.556154 31148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:01.568264 31156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:01.568209 31153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:01.568181 31154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:01.569373 31148 server_base.cc:1034] running on GCE node
I20260924 01:08:01.570188 31148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:01.571738 31148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:01.572971 31148 hybrid_clock.cc:648] HybridClock initialized: now 1790212081572948 us; error 56 us; skew 500 ppm
I20260924 01:08:01.573339 31148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:01 dist-test-slave-5c7l krb5kdc[31135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212081, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:01.576502 31148 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:01.577986 31148 webserver.cc:460] Webserver started at http://127.8.254.126:33553/ using document root <none> and password file <none>
I20260924 01:08:01.578753 31148 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:01.578900 31148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:01.579212 31148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:01.582442 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "6cba1ccf8c6a427f8c52488c677e2a6c"
format_stamp: "Formatted at 2026-09-24 01:08:01 on dist-test-slave-5c7l"
server_key: "c80d91bdc42faca2da316943dfe2a8e0"
server_key_iv: "a3e7733b4bab190a5782e26bc81d7155"
server_key_version: "encryptionkey@0"
I20260924 01:08:01.583257 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "6cba1ccf8c6a427f8c52488c677e2a6c"
format_stamp: "Formatted at 2026-09-24 01:08:01 on dist-test-slave-5c7l"
server_key: "c80d91bdc42faca2da316943dfe2a8e0"
server_key_iv: "a3e7733b4bab190a5782e26bc81d7155"
server_key_version: "encryptionkey@0"
I20260924 01:08:01.588529 31148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:08:01.592054 31163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:01.593330 31148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 01:08:01.593518 31148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "6cba1ccf8c6a427f8c52488c677e2a6c"
format_stamp: "Formatted at 2026-09-24 01:08:01 on dist-test-slave-5c7l"
server_key: "c80d91bdc42faca2da316943dfe2a8e0"
server_key_iv: "a3e7733b4bab190a5782e26bc81d7155"
server_key_version: "encryptionkey@0"
I20260924 01:08:01.593743 31148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:01.625054 31148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:01.629243 31148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:01.629540 31148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:01.665138 31148 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:46587
I20260924 01:08:01.665181 31214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:46587 every 8 connection(s)
I20260924 01:08:01.666757 31148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:01.671005 31215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:01.673969 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31148
I20260924 01:08:01.674175 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:01.674651 9209 external_mini_cluster.cc:1428] Setting key e227bb97ee058688f01b4369f5c882ca
I20260924 01:08:01.683163 31215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c: Bootstrap starting.
I20260924 01:08:01.686697 31215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:01.687971 31215 log.cc:824] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:01.688505 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:01.676611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45877 (local address 127.8.254.126:46587)
0924 01:08:01.677478 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:01.677491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:01.677533 (+ 42us) server_negotiation.cc:407] Connection header received
0924 01:08:01.679341 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:01.679367 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:01.679827 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:01.680254 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:01.680863 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:01.682240 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:01.683720 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:01.684499 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:01.684712 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:01.686014 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:01.686022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:01.686038 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:01.686648 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:01.686741 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:01.686767 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:01.686965 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:01.687248 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 01:08:01.687431 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":375,"threads_started":1}
I20260924 01:08:01.691241 31215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c: No bootstrap required, opened a new log
I20260924 01:08:01.692637 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:01.700069 31215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46587 } }
I20260924 01:08:01.700532 31215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:08:01.700659 31215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cba1ccf8c6a427f8c52488c677e2a6c, State: Initialized, Role: FOLLOWER
I20260924 01:08:01.701270 31215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46587 } }
I20260924 01:08:01.701519 31215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:08:01.701668 31215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:08:01.701876 31215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:01.704319 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:01.697150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46509 (local address 127.8.254.126:46587)
0924 01:08:01.697361 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:01.697372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:01.697753 (+ 381us) server_negotiation.cc:407] Connection header received
0924 01:08:01.697918 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:01.697923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:01.698017 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:01.698129 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:01.698875 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:01.699975 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:01.701617 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:01.702177 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:01.702351 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:01.703532 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:01.703538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:01.703547 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:01.703567 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:01.703625 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:01.703643 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:01.703773 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:01.703963 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 01:08:01.704021 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 01:08:01.704443 31215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46587 } }
I20260924 01:08:01.705032 31215 leader_election.cc:302] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cba1ccf8c6a427f8c52488c677e2a6c; no voters:
I20260924 01:08:01.705788 31215 leader_election.cc:288] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:08:01.705927 31225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:01.706754 31225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [term 1 LEADER]: Becoming Leader. State: Replica: 6cba1ccf8c6a427f8c52488c677e2a6c, State: Running, Role: LEADER
I20260924 01:08:01.707346 31225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46587 } }
I20260924 01:08:01.708114 31215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:01.710860 31227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cba1ccf8c6a427f8c52488c677e2a6c. Latest consensus state: current_term: 1 leader_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46587 } } }
I20260924 01:08:01.711339 31227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:01.712246 31226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba1ccf8c6a427f8c52488c677e2a6c" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 46587 } } }
I20260924 01:08:01.712713 31226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:01.713083 31232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:08:01.724715 31232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:01.733387 31232 catalog_manager.cc:1347] Generated new cluster ID: 97fac9f9a553493daea9384b54342cfd
I20260924 01:08:01.733527 31232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:01.746171 31232 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:01.748067 31232 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:01.760037 31232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cba1ccf8c6a427f8c52488c677e2a6c: Generated new TSK 0
I20260924 01:08:01.760933 31232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:01.770253 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31148
W20260924 01:08:01.771741 31210 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45877
W20260924 01:08:01.772523 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31148: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:08:01.772693 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31148
W20260924 01:08:01.791339 31133 connection.cc:535] client connection to 127.8.254.126:46587 recv error: Network error: failed to read from TLS socket (remote: 127.8.254.126:46587): Connection reset by peer (error 104)
2026-09-24T01:08:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:08:01 dist-test-slave-5c7l krb5kdc[31246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:08:01 dist-test-slave-5c7l krb5kdc[31246](info): set up 2 sockets
Sep 24 01:08:01 dist-test-slave-5c7l krb5kdc[31246](info): commencing operation
krb5kdc: starting...
W20260924 01:08:02.879920 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:03 dist-test-slave-5c7l krb5kdc[31246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:03.153501 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:43531
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:43531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:03.498467 31259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:03.498819 31259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:03.498982 31259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:03.506662 31259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:03.506796 31259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:03.506919 31259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:03.507015 31259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:03.507090 31259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:03.517661 31259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:43531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:03.519142 31259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:03.520484 31259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:03.532997 31264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:03.533244 31265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:03.533320 31267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:03.534325 31259 server_base.cc:1034] running on GCE node
I20260924 01:08:03.535202 31259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:03.536748 31259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:03.537994 31259 hybrid_clock.cc:648] HybridClock initialized: now 1790212083537946 us; error 44 us; skew 500 ppm
I20260924 01:08:03.538436 31259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:03 dist-test-slave-5c7l krb5kdc[31246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:03.552906 31259 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:03.554809 31259 webserver.cc:460] Webserver started at http://127.8.254.126:43341/ using document root <none> and password file <none>
I20260924 01:08:03.555572 31259 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:03.555724 31259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:03.556066 31259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:03.559362 31259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "8c5ff099d1fb42c3938c676355664112"
format_stamp: "Formatted at 2026-09-24 01:08:03 on dist-test-slave-5c7l"
server_key: "cc121b3c38512c0914095cc56df1e43c"
server_key_iv: "a84574c5b5aba0e40e4142ea87deac68"
server_key_version: "encryptionkey@0"
I20260924 01:08:03.560205 31259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "8c5ff099d1fb42c3938c676355664112"
format_stamp: "Formatted at 2026-09-24 01:08:03 on dist-test-slave-5c7l"
server_key: "cc121b3c38512c0914095cc56df1e43c"
server_key_iv: "a84574c5b5aba0e40e4142ea87deac68"
server_key_version: "encryptionkey@0"
I20260924 01:08:03.565764 31259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:08:03.569773 31274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:03.571273 31259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 01:08:03.571478 31259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "8c5ff099d1fb42c3938c676355664112"
format_stamp: "Formatted at 2026-09-24 01:08:03 on dist-test-slave-5c7l"
server_key: "cc121b3c38512c0914095cc56df1e43c"
server_key_iv: "a84574c5b5aba0e40e4142ea87deac68"
server_key_version: "encryptionkey@0"
I20260924 01:08:03.571704 31259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:03.577867 31259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:03.582326 31259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:03.582649 31259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:03.601686 31259 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:43531
I20260924 01:08:03.601711 31325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:43531 every 8 connection(s)
I20260924 01:08:03.603224 31259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:03.607425 31326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:03.609633 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31259
I20260924 01:08:03.609819 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:03.610431 9209 external_mini_cluster.cc:1428] Setting key e6383116127b06233e2376ef47dbce16
I20260924 01:08:03.618544 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:03.612258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40151 (local address 127.8.254.126:43531)
0924 01:08:03.613240 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:03.613252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:03.613301 (+ 49us) server_negotiation.cc:407] Connection header received
0924 01:08:03.614991 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:03.615020 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:03.615392 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:03.615728 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:03.616914 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:03.616920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:03.616933 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:03.616947 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:03.617004 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:03.617025 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:03.617194 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:03.617295 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 01:08:03.617501 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":506,"threads_started":1}
I20260924 01:08:03.622148 31326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112: Bootstrap starting.
I20260924 01:08:03.622604 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:03.626183 31326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:03.627682 31326 log.cc:824] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:03.631165 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:03.628282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46389 (local address 127.8.254.126:43531)
0924 01:08:03.628450 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:03.628454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:03.629061 (+ 607us) server_negotiation.cc:407] Connection header received
0924 01:08:03.629284 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:03.629292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:03.629375 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:03.629478 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:03.630478 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:03.630483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:03.630488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:03.630511 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:03.630563 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:03.630585 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:03.630705 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:03.630811 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 01:08:03.630883 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 01:08:03.631446 31326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112: No bootstrap required, opened a new log
I20260924 01:08:03.639637 31326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5ff099d1fb42c3938c676355664112" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43531 } }
I20260924 01:08:03.640075 31326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:08:03.640196 31326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5ff099d1fb42c3938c676355664112, State: Initialized, Role: FOLLOWER
I20260924 01:08:03.640786 31326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5ff099d1fb42c3938c676355664112" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43531 } }
I20260924 01:08:03.641043 31326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:08:03.641201 31326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:08:03.641364 31326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:03.644196 31326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5ff099d1fb42c3938c676355664112" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43531 } }
I20260924 01:08:03.644734 31326 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c5ff099d1fb42c3938c676355664112; no voters:
I20260924 01:08:03.645340 31326 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:08:03.645507 31336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:03.646224 31336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [term 1 LEADER]: Becoming Leader. State: Replica: 8c5ff099d1fb42c3938c676355664112, State: Running, Role: LEADER
I20260924 01:08:03.646862 31336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5ff099d1fb42c3938c676355664112" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43531 } }
I20260924 01:08:03.647620 31326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:03.650318 31337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c5ff099d1fb42c3938c676355664112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5ff099d1fb42c3938c676355664112" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43531 } } }
I20260924 01:08:03.649889 31338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5ff099d1fb42c3938c676355664112. Latest consensus state: current_term: 1 leader_uuid: "8c5ff099d1fb42c3938c676355664112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5ff099d1fb42c3938c676355664112" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 43531 } } }
I20260924 01:08:03.650902 31337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:03.650942 31338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:03.653550 31343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:08:03.658077 31343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:03.666836 31343 catalog_manager.cc:1347] Generated new cluster ID: e1e6fee5ee374b7e944876be4c57cc9b
I20260924 01:08:03.666968 31343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:03.682471 31343 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:03.683014 31343 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:03.692348 31343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c5ff099d1fb42c3938c676355664112: Generated new TSK 0
I20260924 01:08:03.693440 31343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:03.700534 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31259
W20260924 01:08:03.701737 31321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40151
W20260924 01:08:03.702514 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31259: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:08:03.702672 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31259
2026-09-24T01:08:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:08:03 dist-test-slave-5c7l krb5kdc[31357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:08:03 dist-test-slave-5c7l krb5kdc[31357](info): set up 2 sockets
Sep 24 01:08:03 dist-test-slave-5c7l krb5kdc[31357](info): commencing operation
krb5kdc: starting...
W20260924 01:08:04.824469 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:04 dist-test-slave-5c7l krb5kdc[31357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:05.098564 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:37041
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:05.439690 31370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:05.439991 31370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:05.440193 31370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:05.447551 31370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:05.447753 31370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:05.447875 31370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:05.447973 31370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:05.448056 31370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:05.458307 31370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:37041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:05.459702 31370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:05.461037 31370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:05.472802 31378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:05.472836 31375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:05.472872 31376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:05.473531 31370 server_base.cc:1034] running on GCE node
I20260924 01:08:05.474453 31370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:05.475865 31370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:05.477092 31370 hybrid_clock.cc:648] HybridClock initialized: now 1790212085477053 us; error 48 us; skew 500 ppm
I20260924 01:08:05.477464 31370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:05 dist-test-slave-5c7l krb5kdc[31357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212085, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:05.480691 31370 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:05.482404 31370 webserver.cc:460] Webserver started at http://127.8.254.126:34027/ using document root <none> and password file <none>
I20260924 01:08:05.483104 31370 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:05.483251 31370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:05.483570 31370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:05.486691 31370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "8d065ac91ac94a3095df4c6864b91370"
format_stamp: "Formatted at 2026-09-24 01:08:05 on dist-test-slave-5c7l"
server_key: "932c64a89dcc70c5b31c2932cc0645a7"
server_key_iv: "91b2129acc62ec57e7044ed75b1bff8f"
server_key_version: "encryptionkey@0"
I20260924 01:08:05.487641 31370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "8d065ac91ac94a3095df4c6864b91370"
format_stamp: "Formatted at 2026-09-24 01:08:05 on dist-test-slave-5c7l"
server_key: "932c64a89dcc70c5b31c2932cc0645a7"
server_key_iv: "91b2129acc62ec57e7044ed75b1bff8f"
server_key_version: "encryptionkey@0"
I20260924 01:08:05.493325 31370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 01:08:05.497928 31385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:05.499356 31370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:08:05.499501 31370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "8d065ac91ac94a3095df4c6864b91370"
format_stamp: "Formatted at 2026-09-24 01:08:05 on dist-test-slave-5c7l"
server_key: "932c64a89dcc70c5b31c2932cc0645a7"
server_key_iv: "91b2129acc62ec57e7044ed75b1bff8f"
server_key_version: "encryptionkey@0"
I20260924 01:08:05.499666 31370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:05.526530 31370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:05.530475 31370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:05.530723 31370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:05.548898 31370 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:37041
I20260924 01:08:05.548918 31436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:37041 every 8 connection(s)
I20260924 01:08:05.550401 31370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:05.554380 31437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:05.554977 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31370
I20260924 01:08:05.555159 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:05.555734 9209 external_mini_cluster.cc:1428] Setting key b9064e82b7e65aef99360318e62c6f8d
I20260924 01:08:05.568780 31437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370: Bootstrap starting.
Sep 24 01:08:05 dist-test-slave-5c7l krb5kdc[31357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:08:05.572537 31437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:05.573716 31437 log.cc:824] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:05.576676 31437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370: No bootstrap required, opened a new log
I20260924 01:08:05.579555 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:05.557571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42147 (local address 127.8.254.126:37041)
0924 01:08:05.558393 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:05.558406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:05.558452 (+ 46us) server_negotiation.cc:407] Connection header received
0924 01:08:05.560632 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:05.560655 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:05.561075 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:05.561417 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:05.563263 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:05.564568 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:05.566425 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:05.567099 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:05.567228 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:05.572070 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:05.572078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:05.572095 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:05.572136 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:05.575345 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:05.576065 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:05.576073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:05.576080 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:05.576254 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:05.576656 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:05.576663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:05.576665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:05.577254 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:05.577570 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:05.578049 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 01:08:05.578512 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":376,"threads_started":1}
I20260924 01:08:05.583528 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:05.585311 31437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d065ac91ac94a3095df4c6864b91370" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 37041 } }
I20260924 01:08:05.585680 31437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:08:05.585793 31437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d065ac91ac94a3095df4c6864b91370, State: Initialized, Role: FOLLOWER
I20260924 01:08:05.586467 31437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d065ac91ac94a3095df4c6864b91370" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 37041 } }
I20260924 01:08:05.586849 31437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:08:05.587003 31437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:08:05.587174 31437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:05.589779 31437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d065ac91ac94a3095df4c6864b91370" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 37041 } }
I20260924 01:08:05.590351 31437 leader_election.cc:302] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d065ac91ac94a3095df4c6864b91370; no voters:
I20260924 01:08:05.591053 31437 leader_election.cc:288] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:08:05.591159 31447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:05.592561 31447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [term 1 LEADER]: Becoming Leader. State: Replica: 8d065ac91ac94a3095df4c6864b91370, State: Running, Role: LEADER
I20260924 01:08:05.593075 31447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d065ac91ac94a3095df4c6864b91370" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 37041 } }
I20260924 01:08:05.593218 31437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:05.599900 31448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d065ac91ac94a3095df4c6864b91370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d065ac91ac94a3095df4c6864b91370" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 37041 } } }
I20260924 01:08:05.600332 31448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:05.600891 31449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d065ac91ac94a3095df4c6864b91370. Latest consensus state: current_term: 1 leader_uuid: "8d065ac91ac94a3095df4c6864b91370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d065ac91ac94a3095df4c6864b91370" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 37041 } } }
I20260924 01:08:05.601197 31449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:05.608542 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:05.587989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33209 (local address 127.8.254.126:37041)
0924 01:08:05.588193 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:05.588199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:05.588605 (+ 406us) server_negotiation.cc:407] Connection header received
0924 01:08:05.588748 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:05.588754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:05.588830 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:05.588943 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:05.590405 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:05.591632 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:05.593361 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:05.594068 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:05.594236 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:05.598341 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:05.598348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:05.598354 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:05.598408 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:05.603397 (+ 4989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:05.604583 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:05.604590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:05.604596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:05.604765 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:05.605355 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:05.605363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:05.605369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:05.605898 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:05.607511 (+ 1613us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:05.607700 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 01:08:05.608009 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 01:08:05.613358 31455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:08:05.619372 31455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:05.629128 31455 catalog_manager.cc:1347] Generated new cluster ID: 42322fb4b0da4cf9abb70b60d10bcc86
I20260924 01:08:05.629264 31455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:05.646278 31455 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:05.648269 31455 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:05.672500 31455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d065ac91ac94a3095df4c6864b91370: Generated new TSK 0
I20260924 01:08:05.673434 31455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:05.685856 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31370
I20260924 01:08:05.686924 31432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:05.783658 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31370
2026-09-24T01:08:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2081 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:08:05 dist-test-slave-5c7l krb5kdc[31470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:08:05 dist-test-slave-5c7l krb5kdc[31470](info): set up 2 sockets
Sep 24 01:08:05 dist-test-slave-5c7l krb5kdc[31470](info): commencing operation
krb5kdc: starting...
W20260924 01:08:06.899854 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:07 dist-test-slave-5c7l krb5kdc[31470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:07.177058 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45661
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:07.516924 31483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:07.517254 31483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:07.517411 31483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:07.525023 31483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:07.525141 31483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:07.525189 31483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:07.525290 31483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:07.525408 31483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:07.535566 31483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:07.536993 31483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:07.538365 31483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:07.550836 31488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:07.550901 31491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:07.550899 31489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:07.551695 31483 server_base.cc:1034] running on GCE node
I20260924 01:08:07.552495 31483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:07.553948 31483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:07.555156 31483 hybrid_clock.cc:648] HybridClock initialized: now 1790212087555105 us; error 38 us; skew 500 ppm
I20260924 01:08:07.555490 31483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:07 dist-test-slave-5c7l krb5kdc[31470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212087, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:07.558635 31483 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:07.560150 31483 webserver.cc:460] Webserver started at http://127.8.254.126:43827/ using document root <none> and password file <none>
I20260924 01:08:07.560876 31483 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:07.561028 31483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:07.561353 31483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:07.564597 31483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "b052ad034f714edaa38789ab332b0866"
format_stamp: "Formatted at 2026-09-24 01:08:07 on dist-test-slave-5c7l"
server_key: "e50f3b5d0911536417ac35cab1d6d19e"
server_key_iv: "3cc933a179b0a82be22ad34c3a1003d2"
server_key_version: "encryptionkey@0"
I20260924 01:08:07.565407 31483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "b052ad034f714edaa38789ab332b0866"
format_stamp: "Formatted at 2026-09-24 01:08:07 on dist-test-slave-5c7l"
server_key: "e50f3b5d0911536417ac35cab1d6d19e"
server_key_iv: "3cc933a179b0a82be22ad34c3a1003d2"
server_key_version: "encryptionkey@0"
I20260924 01:08:07.570825 31483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:08:07.574486 31498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:07.575824 31483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:08:07.576016 31483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "b052ad034f714edaa38789ab332b0866"
format_stamp: "Formatted at 2026-09-24 01:08:07 on dist-test-slave-5c7l"
server_key: "e50f3b5d0911536417ac35cab1d6d19e"
server_key_iv: "3cc933a179b0a82be22ad34c3a1003d2"
server_key_version: "encryptionkey@0"
I20260924 01:08:07.576224 31483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:07.589962 31483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:07.594260 31483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:07.594581 31483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:07.613328 31483 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45661
I20260924 01:08:07.613363 31549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45661 every 8 connection(s)
I20260924 01:08:07.614921 31483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:07.619068 31550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:07.623083 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31483
I20260924 01:08:07.623261 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:07.623720 9209 external_mini_cluster.cc:1428] Setting key cf251177233b794e3d861fe09bfcfbb4
I20260924 01:08:07.632261 31550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866: Bootstrap starting.
I20260924 01:08:07.636102 31550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:07.637346 31550 log.cc:824] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:07.637969 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:07.625583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43833 (local address 127.8.254.126:45661)
0924 01:08:07.626449 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:07.626462 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:07.626505 (+ 43us) server_negotiation.cc:407] Connection header received
0924 01:08:07.628175 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:07.628199 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:07.628667 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:07.629052 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:07.629673 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:07.631112 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:07.632993 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:07.633703 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:07.633858 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:07.635287 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:07.635295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:07.635311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:07.635890 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:07.635971 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:07.636002 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:07.636107 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:07.636429 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 01:08:07.636750 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":371,"threads_started":1}
I20260924 01:08:07.640398 31550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866: No bootstrap required, opened a new log
I20260924 01:08:07.641817 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:07.650982 31550 raft_consensus.cc:348] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b052ad034f714edaa38789ab332b0866" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45661 } }
I20260924 01:08:07.651400 31550 raft_consensus.cc:374] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:08:07.651535 31550 raft_consensus.cc:729] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b052ad034f714edaa38789ab332b0866, State: Initialized, Role: FOLLOWER
I20260924 01:08:07.652231 31550 consensus_queue.cc:260] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b052ad034f714edaa38789ab332b0866" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45661 } }
I20260924 01:08:07.652504 31550 raft_consensus.cc:388] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:08:07.652657 31550 raft_consensus.cc:482] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:08:07.652845 31550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:07.654399 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:07.647339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49075 (local address 127.8.254.126:45661)
0924 01:08:07.647563 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:07.647574 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:07.648011 (+ 437us) server_negotiation.cc:407] Connection header received
0924 01:08:07.648180 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:07.648186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:07.648280 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:07.648398 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:07.648902 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:07.650050 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:07.651537 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:07.652147 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:07.652293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:07.653576 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:07.653587 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:07.653590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:07.653635 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:07.653678 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:07.653698 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:07.653809 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:07.653951 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 01:08:07.654054 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 01:08:07.655557 31550 raft_consensus.cc:504] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b052ad034f714edaa38789ab332b0866" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45661 } }
I20260924 01:08:07.656157 31550 leader_election.cc:302] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b052ad034f714edaa38789ab332b0866; no voters:
I20260924 01:08:07.656713 31550 leader_election.cc:288] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:08:07.656881 31560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:07.657615 31560 raft_consensus.cc:686] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [term 1 LEADER]: Becoming Leader. State: Replica: b052ad034f714edaa38789ab332b0866, State: Running, Role: LEADER
I20260924 01:08:07.658245 31560 consensus_queue.cc:237] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b052ad034f714edaa38789ab332b0866" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45661 } }
I20260924 01:08:07.658995 31550 sys_catalog.cc:564] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:07.661501 31561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b052ad034f714edaa38789ab332b0866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b052ad034f714edaa38789ab332b0866" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45661 } } }
I20260924 01:08:07.661870 31561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:07.663282 31562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b052ad034f714edaa38789ab332b0866. Latest consensus state: current_term: 1 leader_uuid: "b052ad034f714edaa38789ab332b0866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b052ad034f714edaa38789ab332b0866" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45661 } } }
I20260924 01:08:07.663682 31562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:07.670627 31565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:08:07.676194 31565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:07.686002 31565 catalog_manager.cc:1347] Generated new cluster ID: ef31cd3e6d8f483c9484d5365e641b4b
I20260924 01:08:07.686157 31565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:07.700651 31565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:07.702642 31565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:07.712152 31565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b052ad034f714edaa38789ab332b0866: Generated new TSK 0
I20260924 01:08:07.713280 31565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:07.725607 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31483
W20260924 01:08:07.726984 31545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43833
W20260924 01:08:07.727757 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:08:07.727931 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31483
2026-09-24T01:08:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:08:07 dist-test-slave-5c7l krb5kdc[31581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:08:07 dist-test-slave-5c7l krb5kdc[31581](info): set up 2 sockets
Sep 24 01:08:07 dist-test-slave-5c7l krb5kdc[31581](info): commencing operation
krb5kdc: starting...
W20260924 01:08:08.843969 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:09 dist-test-slave-5c7l krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:09.109774 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45471
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:09.445621 31594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:09.445988 31594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:09.446153 31594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:09.453500 31594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:09.453689 31594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:09.453794 31594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:09.453902 31594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:09.453977 31594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:09.464349 31594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:45471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:09.465708 31594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:09.467075 31594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:09.479182 31602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:09.479249 31599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:09.479212 31600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:09.480337 31594 server_base.cc:1034] running on GCE node
I20260924 01:08:09.481102 31594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:09.482556 31594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:09.483770 31594 hybrid_clock.cc:648] HybridClock initialized: now 1790212089483728 us; error 31 us; skew 500 ppm
I20260924 01:08:09.484133 31594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:09 dist-test-slave-5c7l krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:09.493875 31594 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:09.496078 31594 webserver.cc:460] Webserver started at http://127.8.254.126:38409/ using document root <none> and password file <none>
I20260924 01:08:09.496806 31594 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:09.496956 31594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:09.497285 31594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:09.500456 31594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "4f62cd8f892e4a2288c53d42941fc8c1"
format_stamp: "Formatted at 2026-09-24 01:08:09 on dist-test-slave-5c7l"
server_key: "7f7c6e2c61180b7c42e1a88ba48e564c"
server_key_iv: "b2a57a800722f1f0cca9a329b84c5095"
server_key_version: "encryptionkey@0"
I20260924 01:08:09.501264 31594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "4f62cd8f892e4a2288c53d42941fc8c1"
format_stamp: "Formatted at 2026-09-24 01:08:09 on dist-test-slave-5c7l"
server_key: "7f7c6e2c61180b7c42e1a88ba48e564c"
server_key_iv: "b2a57a800722f1f0cca9a329b84c5095"
server_key_version: "encryptionkey@0"
I20260924 01:08:09.506580 31594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 01:08:09.510032 31609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:09.511430 31594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:08:09.511626 31594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "4f62cd8f892e4a2288c53d42941fc8c1"
format_stamp: "Formatted at 2026-09-24 01:08:09 on dist-test-slave-5c7l"
server_key: "7f7c6e2c61180b7c42e1a88ba48e564c"
server_key_iv: "b2a57a800722f1f0cca9a329b84c5095"
server_key_version: "encryptionkey@0"
I20260924 01:08:09.511866 31594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:09.517735 31594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:09.522089 31594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:09.522454 31594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:09.541603 31594 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:45471
I20260924 01:08:09.541635 31660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:45471 every 8 connection(s)
I20260924 01:08:09.543164 31594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:09.545953 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31594
I20260924 01:08:09.546144 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:09.546694 9209 external_mini_cluster.cc:1428] Setting key 555644064b32215668cb82a18ea47c66
I20260924 01:08:09.547360 31661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:09.554778 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:09.548568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36133 (local address 127.8.254.126:45471)
0924 01:08:09.549383 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:09.549394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:09.549435 (+ 41us) server_negotiation.cc:407] Connection header received
0924 01:08:09.551062 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:09.551084 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:09.551455 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:09.551883 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:09.553016 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:09.553025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:09.553039 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:09.553055 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:09.553120 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:09.553139 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:09.553396 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:09.553554 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 01:08:09.553753 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":344,"threads_started":1}
I20260924 01:08:09.558635 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:09.560820 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1: Bootstrap starting.
I20260924 01:08:09.564752 31661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:09.565828 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:09.563140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47373 (local address 127.8.254.126:45471)
0924 01:08:09.563321 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:09.563327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:09.563649 (+ 322us) server_negotiation.cc:407] Connection header received
0924 01:08:09.563802 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:09.563809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:09.563876 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:09.563995 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:09.565174 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:09.565179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:09.565184 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 01:08:09.565203 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:09.565261 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 01:08:09.565285 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:09.565418 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:09.565491 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 01:08:09.565548 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:08:09.566252 31661 log.cc:824] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:09.569257 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1: No bootstrap required, opened a new log
I20260924 01:08:09.576833 31661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45471 } }
I20260924 01:08:09.577241 31661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 01:08:09.577371 31661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f62cd8f892e4a2288c53d42941fc8c1, State: Initialized, Role: FOLLOWER
I20260924 01:08:09.578032 31661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45471 } }
I20260924 01:08:09.578329 31661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 01:08:09.578495 31661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 01:08:09.578672 31661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:09.581259 31661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45471 } }
I20260924 01:08:09.581768 31661 leader_election.cc:302] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f62cd8f892e4a2288c53d42941fc8c1; no voters:
I20260924 01:08:09.582427 31661 leader_election.cc:288] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 01:08:09.582579 31671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:09.583315 31671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [term 1 LEADER]: Becoming Leader. State: Replica: 4f62cd8f892e4a2288c53d42941fc8c1, State: Running, Role: LEADER
I20260924 01:08:09.583828 31671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45471 } }
I20260924 01:08:09.584705 31661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:09.587116 31673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f62cd8f892e4a2288c53d42941fc8c1. Latest consensus state: current_term: 1 leader_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45471 } } }
I20260924 01:08:09.587196 31672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f62cd8f892e4a2288c53d42941fc8c1" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 45471 } } }
I20260924 01:08:09.587728 31673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:09.587795 31672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:09.590108 31677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:08:09.595978 31677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:09.604954 31677 catalog_manager.cc:1347] Generated new cluster ID: d39d6f39ca294d9db95608d8dfd188ef
I20260924 01:08:09.605098 31677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:09.636842 31677 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:09.637387 31677 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:09.660733 31677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f62cd8f892e4a2288c53d42941fc8c1: Generated new TSK 0
I20260924 01:08:09.661978 31677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:09.675730 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31594
W20260924 01:08:09.676968 31656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36133
W20260924 01:08:09.677721 9209 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31594: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 01:08:09.677874 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31594
2026-09-24T01:08:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1948 ms)
[----------] 18 tests from AuthTokenIssuingTest (35694 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:08:09 dist-test-slave-5c7l krb5kdc[31692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:08:09 dist-test-slave-5c7l krb5kdc[31692](info): set up 2 sockets
Sep 24 01:08:09 dist-test-slave-5c7l krb5kdc[31692](info): commencing operation
krb5kdc: starting...
W20260924 01:08:10.796140 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:10 dist-test-slave-5c7l krb5kdc[31692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:11.071405 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:35851
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:35851,127.8.254.125:44663,127.8.254.124:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:11.407362 31705 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 01:08:11.407727 31705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:11.407867 31705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:11.408012 31705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:11.415532 31705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:11.415625 31705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:11.415753 31705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:11.415822 31705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:11.415908 31705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:11.426057 31705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:35851,127.8.254.125:44663,127.8.254.124:38053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:35851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:11.427465 31705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:11.428831 31705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:11.441272 31710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:11.441318 31713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:11.441318 31711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:11.442154 31705 server_base.cc:1034] running on GCE node
I20260924 01:08:11.443029 31705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:11.444450 31705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:11.445696 31705 hybrid_clock.cc:648] HybridClock initialized: now 1790212091445627 us; error 41 us; skew 500 ppm
I20260924 01:08:11.446072 31705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:11 dist-test-slave-5c7l krb5kdc[31692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:11.449551 31705 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:11.451107 31705 webserver.cc:460] Webserver started at http://127.8.254.126:33711/ using document root <none> and password file <none>
I20260924 01:08:11.451825 31705 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:11.451964 31705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:11.452304 31705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:11.455436 31705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "6f2e63de21944594b76ac66d0eea828e"
format_stamp: "Formatted at 2026-09-24 01:08:11 on dist-test-slave-5c7l"
server_key: "7c1c3ed137e95aaa788bd6e7f438c3ef"
server_key_iv: "fac6cf2785165caa206a1383078b8216"
server_key_version: "encryptionkey@0"
I20260924 01:08:11.456351 31705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "6f2e63de21944594b76ac66d0eea828e"
format_stamp: "Formatted at 2026-09-24 01:08:11 on dist-test-slave-5c7l"
server_key: "7c1c3ed137e95aaa788bd6e7f438c3ef"
server_key_iv: "fac6cf2785165caa206a1383078b8216"
server_key_version: "encryptionkey@0"
I20260924 01:08:11.461758 31705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 01:08:11.465431 31720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:11.467005 31705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:08:11.467211 31705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "6f2e63de21944594b76ac66d0eea828e"
format_stamp: "Formatted at 2026-09-24 01:08:11 on dist-test-slave-5c7l"
server_key: "7c1c3ed137e95aaa788bd6e7f438c3ef"
server_key_iv: "fac6cf2785165caa206a1383078b8216"
server_key_version: "encryptionkey@0"
I20260924 01:08:11.467445 31705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:11.480423 31705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:11.484763 31705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:11.485060 31705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:11.508445 31705 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:35851
I20260924 01:08:11.508452 31771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:35851 every 8 connection(s)
I20260924 01:08:11.509876 31705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:11.513772 31772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:11.517148 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31705
I20260924 01:08:11.517373 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:11.518030 9209 external_mini_cluster.cc:1428] Setting key 563614fb1dc3708052a1fccdde12e9c5
I20260924 01:08:11.520874 31772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:11 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.125@KRBTEST.COM" created.
I20260924 01:08:11.544992 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.524355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:35851 (local address 127.0.0.1:33786)
0924 01:08:11.526051 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.526089 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:11.527948 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:11.528865 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:11.528872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:11.529608 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:11.529945 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:11.529967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:11.531832 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:11.531837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:11.532917 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:11.532930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:11.533904 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:11.533911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:11.534078 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:11.534893 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:11.534916 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:11.537474 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:11.540810 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:11.540822 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:11.540835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:11.541298 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:11.541700 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:11.541708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:11.541713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:11.541920 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:11.542856 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:11.542864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:11.543102 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 01:08:11.543419 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 01:08:11.543910 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":1151,"threads_started":1}
I20260924 01:08:11.545035 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.524355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.8.254.126:35851)
0924 01:08:11.525521 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:11.525539 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:11.526248 (+ 709us) server_negotiation.cc:407] Connection header received
0924 01:08:11.528359 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:11.528365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:11.528619 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:11.528846 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:11.530241 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:11.531659 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:11.533064 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:11.533652 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:11.533883 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:11.537718 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:11.537728 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:11.537742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:11.537772 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:11.540657 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:11.541429 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:11.541444 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:11.541446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:11.541588 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:11.542046 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:11.542052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:11.542061 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:11.542657 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:11.542921 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:11.543592 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 01:08:11.544118 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":902,"spinlock_wait_cycles":43904,"thread_start_us":578,"threads_started":1}
I20260924 01:08:11.548228 31772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:11.549432 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.549017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44104)
0924 01:08:11.549199 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.549321 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 01:08:11.549793 31722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
I20260924 01:08:11.551445 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.551034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44106)
0924 01:08:11.551222 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.551298 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 01:08:11.551777 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:44663: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab.
Entry for principal kudu/127.8.254.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab.
I20260924 01:08:11.575870 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:44663
--webserver_interface=127.8.254.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab
--principal=kudu/127.8.254.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:35851,127.8.254.125:44663,127.8.254.124:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:08:11.601153 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } attempt: 1
I20260924 01:08:11.602444 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.601939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44110)
0924 01:08:11.602131 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.602294 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 01:08:11.603449 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.603106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44124)
0924 01:08:11.603254 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.603328 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 01:08:11.603650 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:44663: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
I20260924 01:08:11.671977 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } attempt: 2
I20260924 01:08:11.673144 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.672657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44132)
0924 01:08:11.672921 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.673006 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 01:08:11.674276 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.673841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44134)
0924 01:08:11.673996 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.674084 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 01:08:11.674491 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:44663: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
I20260924 01:08:11.765786 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } attempt: 3
I20260924 01:08:11.767257 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.766779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44140)
0924 01:08:11.766988 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.767095 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 01:08:11.768380 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.767971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44146)
0924 01:08:11.768152 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.768235 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:08:11.768661 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:44663: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
I20260924 01:08:11.911948 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } attempt: 4
I20260924 01:08:11.913239 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.912676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44148)
0924 01:08:11.912993 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.913080 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260924 01:08:11.914384 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:11.913947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44156)
0924 01:08:11.914101 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:11.914250 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 01:08:11.914613 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:44663: Network error: Client connection negotiation failed: client connection to 127.8.254.125:44663: connect: Connection refused (error 111)
W20260924 01:08:11.915438 31779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 01:08:11.915807 31779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:11.915923 31779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:11.916021 31779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:11.924078 31779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:11.924177 31779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:11.924268 31779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:11.924360 31779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:11.924471 31779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:11.934916 31779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:35851,127.8.254.125:44663,127.8.254.124:38053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab
--principal=kudu/127.8.254.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.254.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:11.936336 31779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:11.937674 31779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:11.949790 31784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:11.949838 31785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:11.950109 31787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:11.951635 31779 server_base.cc:1034] running on GCE node
I20260924 01:08:11.952487 31779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:11.954001 31779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:11.955286 31779 hybrid_clock.cc:648] HybridClock initialized: now 1790212091955172 us; error 60 us; skew 500 ppm
I20260924 01:08:11.955942 31779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:11 dist-test-slave-5c7l krb5kdc[31692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:11.959594 31779 init.cc:377] Logged in from keytab as kudu/127.8.254.125@KRBTEST.COM (short username kudu)
I20260924 01:08:11.961282 31779 webserver.cc:460] Webserver started at http://127.8.254.125:42449/ using document root <none> and password file <none>
I20260924 01:08:11.962016 31779 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:11.962153 31779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:11.962502 31779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:11.965688 31779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data/instance:
uuid: "b5468755f4a6474b8f825ba969137c0b"
format_stamp: "Formatted at 2026-09-24 01:08:11 on dist-test-slave-5c7l"
server_key: "e24a982fdf90dc96ac9d588989d11018"
server_key_iv: "0d3637efda684522a494d0533bdcb0c7"
server_key_version: "encryptionkey@0"
I20260924 01:08:11.966535 31779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal/instance:
uuid: "b5468755f4a6474b8f825ba969137c0b"
format_stamp: "Formatted at 2026-09-24 01:08:11 on dist-test-slave-5c7l"
server_key: "e24a982fdf90dc96ac9d588989d11018"
server_key_iv: "0d3637efda684522a494d0533bdcb0c7"
server_key_version: "encryptionkey@0"
I20260924 01:08:11.971947 31779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 01:08:11.975586 31794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:11.976976 31779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:08:11.977164 31779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal
uuid: "b5468755f4a6474b8f825ba969137c0b"
format_stamp: "Formatted at 2026-09-24 01:08:11 on dist-test-slave-5c7l"
server_key: "e24a982fdf90dc96ac9d588989d11018"
server_key_iv: "0d3637efda684522a494d0533bdcb0c7"
server_key_version: "encryptionkey@0"
I20260924 01:08:11.977386 31779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:12.006309 31779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:12.010784 31779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:12.011111 31779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:12.031050 31779 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.125:44663
I20260924 01:08:12.031057 31845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.125:44663 every 8 connection(s)
I20260924 01:08:12.032449 31779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
I20260924 01:08:12.036471 31846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:12.042017 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31779
I20260924 01:08:12.042244 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-1/wal/instance
I20260924 01:08:12.042802 9209 external_mini_cluster.cc:1428] Setting key c860b205f5baf6bc86b772a3a3fb3a32
I20260924 01:08:12.043646 31846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:08:12.069835 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.047435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.8.254.126:35851)
0924 01:08:12.048000 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.048007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.049059 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 01:08:12.052111 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.052116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.052191 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.052292 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.054404 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.055674 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.057223 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.057969 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.058128 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.061927 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.061936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.061944 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.062005 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.066030 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.066997 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.067007 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.067013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.067182 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.067699 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.067706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.067711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.068158 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.068347 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.069126 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.069394 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
I20260924 01:08:12.070596 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.048019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:35851 (local address 127.0.0.1:33790)
0924 01:08:12.048863 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.048914 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.051776 (+ 2862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.052590 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.052605 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.053459 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.054111 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.054130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.055843 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.055847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.057062 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.057075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.058251 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.058260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.058482 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.059268 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.059295 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.061637 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.066266 (+ 4629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.066280 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.066295 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.066839 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.067323 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.067329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.067332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.067562 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.068308 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.068314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.068711 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.068954 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.069375 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for kudu/127.8.254.124@KRBTEST.COM; defaulting to no policy
I20260924 01:08:12.072619 31846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.124@KRBTEST.COM" created.
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:08:12.087929 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.073646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.8.254.125:44663)
0924 01:08:12.074352 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.074358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.074396 (+ 38us) server_negotiation.cc:407] Connection header received
0924 01:08:12.074443 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.074449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.074688 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.074845 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.075567 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.076613 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.077715 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.078314 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.078440 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.081697 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.081713 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.081734 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.081778 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.084829 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.085671 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.085679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.085681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.085803 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.086300 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.086304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.086307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.086653 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.086813 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.087257 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.087588 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":497,"thread_start_us":365,"threads_started":1}
I20260924 01:08:12.087955 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.073429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44164)
0924 01:08:12.073593 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.073623 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.073840 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.074827 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.074832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.075268 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.075396 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.075404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.076744 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.076752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.077571 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.077581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.078450 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.078455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.078672 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.079299 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.079316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.081436 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.084973 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.084981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.084987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.085541 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.085942 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.085950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.085956 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.086130 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.086811 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.086819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.086990 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.087150 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.087470 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 01:08:12.090693 31846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:12.092008 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.091431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43696)
0924 01:08:12.091636 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.091807 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 01:08:12.092314 31797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
I20260924 01:08:12.094055 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.093595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43702)
0924 01:08:12.093811 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.093909 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 01:08:12.094302 31846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab.
Entry for principal kudu/127.8.254.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab.
I20260924 01:08:12.109658 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:38053
--webserver_interface=127.8.254.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab
--principal=kudu/127.8.254.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:35851,127.8.254.125:44663,127.8.254.124:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:08:12.143697 31846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 1
I20260924 01:08:12.144943 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.144423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43710)
0924 01:08:12.144636 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.144761 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 01:08:12.145991 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.145609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43724)
0924 01:08:12.145778 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.145867 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:08:12.146240 31846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
I20260924 01:08:12.213888 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } attempt: 5
I20260924 01:08:12.214452 31846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 2
I20260924 01:08:12.215581 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.215039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43740)
0924 01:08:12.215274 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.215378 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 01:08:12.216687 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.216295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43746)
0924 01:08:12.216475 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.216548 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 01:08:12.216924 31846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:08:12.228873 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.214675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44170)
0924 01:08:12.214986 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.215007 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.215145 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.215542 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.215549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.216074 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.216275 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.216285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.217749 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.217753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.218698 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.218711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.219676 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.219681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.219833 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.220460 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.220478 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.222776 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.226505 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.226510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.226512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.226907 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.227319 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.227322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.227324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.227457 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.228113 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.228116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.228257 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.228371 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.228612 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 01:08:12.229038 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.214751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.8.254.125:44663)
0924 01:08:12.214986 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.214993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.215075 (+ 82us) server_negotiation.cc:407] Connection header received
0924 01:08:12.215279 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.215285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.215365 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.215476 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.216463 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.217603 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.218849 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.219530 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.219659 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.222994 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.223000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.223005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.223045 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.226360 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.227034 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.227040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.227043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.227206 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.227577 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.227584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.227589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.227998 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.228165 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.228511 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.228744 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 01:08:12.230186 31772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:12.231300 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.230905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43754)
0924 01:08:12.231078 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.231167 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 01:08:12.232200 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.231889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43766)
0924 01:08:12.232027 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.232100 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 01:08:12.232398 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
I20260924 01:08:12.283609 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 1
I20260924 01:08:12.284739 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.284259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43778)
0924 01:08:12.284475 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.284566 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 01:08:12.285703 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.285382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43790)
0924 01:08:12.285525 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.285598 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 01:08:12.285916 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
I20260924 01:08:12.308264 31846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 3
I20260924 01:08:12.309643 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.308963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43800)
0924 01:08:12.309260 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.309408 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 01:08:12.310986 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.310409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43802)
0924 01:08:12.310682 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.310797 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260924 01:08:12.311254 31846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
I20260924 01:08:12.354133 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 2
I20260924 01:08:12.355389 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.354883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43808)
0924 01:08:12.355103 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.355207 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 01:08:12.356472 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.356149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43814)
0924 01:08:12.356294 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.356374 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 01:08:12.356755 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
W20260924 01:08:12.450965 31854 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 01:08:12.451320 31854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:12.451455 31854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:12.451577 31854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 01:08:12.454492 31846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 4
I20260924 01:08:12.455721 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.455150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43830)
0924 01:08:12.455412 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.455520 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 01:08:12.456728 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.456371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43838)
0924 01:08:12.456522 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.456603 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 01:08:12.456959 31846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
W20260924 01:08:12.459928 31854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:12.460012 31854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:12.460070 31854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:12.460124 31854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:12.460186 31854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:12.462965 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 3
I20260924 01:08:12.464031 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.463604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43852)
0924 01:08:12.463789 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.463876 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 01:08:12.465037 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.464668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43860)
0924 01:08:12.464862 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.464941 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 01:08:12.465292 31772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:38053: Network error: Client connection negotiation failed: client connection to 127.8.254.124:38053: connect: Connection refused (error 111)
I20260924 01:08:12.470860 31854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:35851,127.8.254.125:44663,127.8.254.124:38053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab
--principal=kudu/127.8.254.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:38053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.254.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:12.472270 31854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:12.473549 31854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:12.485700 31859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:12.485724 31860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:12.486071 31862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:12.486543 31854 server_base.cc:1034] running on GCE node
I20260924 01:08:12.487390 31854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:12.488775 31854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:12.490024 31854 hybrid_clock.cc:648] HybridClock initialized: now 1790212092489949 us; error 52 us; skew 500 ppm
I20260924 01:08:12.490408 31854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212092, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:12.493587 31854 init.cc:377] Logged in from keytab as kudu/127.8.254.124@KRBTEST.COM (short username kudu)
I20260924 01:08:12.495255 31854 webserver.cc:460] Webserver started at http://127.8.254.124:40537/ using document root <none> and password file <none>
I20260924 01:08:12.495959 31854 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:12.496086 31854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:12.496425 31854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:12.499450 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data/instance:
uuid: "7150988e5d7e48879d62ea1469e804dd"
format_stamp: "Formatted at 2026-09-24 01:08:12 on dist-test-slave-5c7l"
server_key: "cc984f2226c4a8768bba056aef70ef4b"
server_key_iv: "91713f45b805e75e9b59ea43f2ef8849"
server_key_version: "encryptionkey@0"
I20260924 01:08:12.500226 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal/instance:
uuid: "7150988e5d7e48879d62ea1469e804dd"
format_stamp: "Formatted at 2026-09-24 01:08:12 on dist-test-slave-5c7l"
server_key: "cc984f2226c4a8768bba056aef70ef4b"
server_key_iv: "91713f45b805e75e9b59ea43f2ef8849"
server_key_version: "encryptionkey@0"
I20260924 01:08:12.505966 31854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 01:08:12.510133 31869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:12.511740 31854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 01:08:12.511937 31854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal
uuid: "7150988e5d7e48879d62ea1469e804dd"
format_stamp: "Formatted at 2026-09-24 01:08:12 on dist-test-slave-5c7l"
server_key: "cc984f2226c4a8768bba056aef70ef4b"
server_key_iv: "91713f45b805e75e9b59ea43f2ef8849"
server_key_version: "encryptionkey@0"
I20260924 01:08:12.512197 31854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:12.525516 31854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:12.529999 31854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:12.530304 31854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:12.548924 31854 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.124:38053
I20260924 01:08:12.548931 31920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.124:38053 every 8 connection(s)
I20260924 01:08:12.550426 31854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
I20260924 01:08:12.554052 31921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:12.555310 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 31854
I20260924 01:08:12.555485 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790211722208042-9209-0/minicluster-data/master-2/wal/instance
I20260924 01:08:12.556048 9209 external_mini_cluster.cc:1428] Setting key e6b265080cee825ca1902f40c55ac561
I20260924 01:08:12.563942 31921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212092, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:08:12.580797 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.558024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.8.254.124:38053)
0924 01:08:12.558934 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.558946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.558986 (+ 40us) server_negotiation.cc:407] Connection header received
0924 01:08:12.560609 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.560634 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.561075 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.561429 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.563270 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.564595 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.566396 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.567602 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.567800 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.572608 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.572618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.572632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.572690 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.576299 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.576955 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.576962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.576970 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.577196 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.577582 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.577587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.577592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.578390 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.578724 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.579038 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.579500 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":459,"threads_started":1}
I20260924 01:08:12.582324 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.566954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:35851 (local address 127.0.0.1:33806)
0924 01:08:12.567706 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.567760 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.567999 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.568424 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.568429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.569332 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.569694 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.569709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.570898 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.570903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.572071 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.572084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.572951 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.572956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.573102 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.573793 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.573823 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.576094 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.579693 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.579702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.579706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.580091 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.580488 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.580491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.580493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.580606 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.581255 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.581265 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.581452 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.581598 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.581802 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":355,"threads_started":1}
I20260924 01:08:12.582368 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.566986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.8.254.126:35851)
0924 01:08:12.567201 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.567210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.567852 (+ 642us) server_negotiation.cc:407] Connection header received
0924 01:08:12.568144 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.568148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.568227 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.568306 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.569875 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.570727 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.572216 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.572765 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.572921 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.576302 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.576309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.576313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.576357 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.579567 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.580212 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.580218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.580221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.580376 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.580711 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.580716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.580720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.581109 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.581262 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.581725 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.581966 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 01:08:12.583997 31921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212092, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:08:12.597035 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.584845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:44663 (local address 127.0.0.1:44174)
0924 01:08:12.585117 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.585130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.585244 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.585575 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.585579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.585925 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.586051 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.586059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.587025 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.587032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.587890 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.587901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.588599 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.588604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.588791 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.589456 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.589473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.591447 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.594713 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.594718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.594723 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.595084 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.595465 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.595470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.595474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.595612 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.596104 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.596109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.596264 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.596401 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.596674 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 01:08:12.597066 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.584830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.8.254.125:44663)
0924 01:08:12.585012 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.585019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.585198 (+ 179us) server_negotiation.cc:407] Connection header received
0924 01:08:12.585376 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.585381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.585456 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.585557 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.586221 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.586894 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.588026 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.588452 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.588573 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.591630 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.591635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.591638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.591670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.594581 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.595199 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.595222 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.595225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.595354 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.595717 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.595723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.595726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.595991 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.596141 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.596487 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.596694 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:08:12.598141 31921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:12.598474 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.584165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.8.254.126:35851)
0924 01:08:12.584326 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.584331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.584429 (+ 98us) server_negotiation.cc:407] Connection header received
0924 01:08:12.584626 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.584631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.584707 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.584818 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.586014 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.586775 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.587988 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.588410 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.588544 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.592576 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.592581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.592584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.592614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.595211 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.595878 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.595885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.595888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.596017 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.596387 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.596392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.596396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.596653 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.596967 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.597887 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.598112 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212092, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:08:12.611739 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.599060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43878)
0924 01:08:12.599216 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.599240 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.599327 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.599623 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.599627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.600059 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.600249 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.600261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.601358 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.601365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.602560 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.602573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.603376 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.603383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.603594 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.604141 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.604165 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.606426 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.609350 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.609353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.609356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.609703 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.610116 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.610119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.610121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.610314 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.610938 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.610944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.611126 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.611235 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.611473 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 01:08:12.612008 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.599167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.8.254.124:38053)
0924 01:08:12.599337 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.599353 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.599373 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:08:12.599422 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.599427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.599499 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.599608 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.600392 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.601222 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.602690 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.603211 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.603302 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.606578 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.606584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.606590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.606622 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.609185 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.609829 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.609836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.609841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.609965 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.610436 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.610441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.610445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.610814 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.610941 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.611389 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.611627 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 01:08:12.615482 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.600168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.8.254.125:44663)
0924 01:08:12.600313 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.600316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.600351 (+ 35us) server_negotiation.cc:407] Connection header received
0924 01:08:12.600516 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.600521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.600587 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.600684 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.601975 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.602801 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.604337 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.604717 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.604803 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.609334 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.609342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.609347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.609383 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.612737 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.613528 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.613533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.613537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.613684 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.614057 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.614062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.614065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.614398 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.614663 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.614939 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.615161 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 01:08:12.617326 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: Bootstrap starting.
I20260924 01:08:12.620522 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:12.621266 31921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:12.622612 31921 log.cc:824] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:12.626016 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: No bootstrap required, opened a new log
I20260924 01:08:12.623471 31747 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6f2e63de21944594b76ac66d0eea828e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33816
I20260924 01:08:12.637176 31921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.637655 31921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7150988e5d7e48879d62ea1469e804dd, State: Initialized, Role: FOLLOWER
I20260924 01:08:12.638271 31921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.639329 31927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.639663 31927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.640745 31921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:12.642546 31772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 4
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
W20260924 01:08:12.652325 31938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:08:12.652505 31938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:08:12.660496 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.643448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.8.254.124:38053)
0924 01:08:12.643720 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.643732 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.643772 (+ 40us) server_negotiation.cc:407] Connection header received
0924 01:08:12.643993 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.643998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.644081 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.644197 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.645368 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.646139 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.647989 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.648715 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.648892 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.653015 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.653022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.653027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.653072 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.657309 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.658086 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.658094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.658098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.658285 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.658762 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.658767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.658771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.659253 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.659417 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.659880 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.660111 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 01:08:12.660486 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.643381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43882)
0924 01:08:12.643690 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.643713 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.643850 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.644281 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.644290 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.644915 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.645155 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.645174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.646398 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.646404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.647826 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.647842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.648888 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.648896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.649094 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.649796 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.649823 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.652764 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.657491 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.657497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.657501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.657927 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.658430 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.658438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.658442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.658631 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.659416 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.659420 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.659592 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.659746 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.660082 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 01:08:12.648947 31854 master_runner.cc:186] Error getting master registration for 127.8.254.126:35851: OK, instance_id { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" instance_seqno: 1790212091502899 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 01:08:12.660976 31854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 01:08:12.666975 31772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e: Bootstrap starting.
I20260924 01:08:12.671545 31772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:12.673007 31772 log.cc:824] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:12.676710 31772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e: No bootstrap required, opened a new log
I20260924 01:08:12.684906 31772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.685391 31772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2e63de21944594b76ac66d0eea828e, State: Initialized, Role: FOLLOWER
I20260924 01:08:12.685966 31772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.687091 31941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.687543 31941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.688522 31772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:12.688576 31747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 01:08:12.688799 31747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:12.691751 31747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.693282 31747 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [CANDIDATE]: Term 1 election: Requested vote from peers b5468755f4a6474b8f825ba969137c0b (127.8.254.125:44663), 7150988e5d7e48879d62ea1469e804dd (127.8.254.124:38053)
I20260924 01:08:12.694782 31896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f2e63de21944594b76ac66d0eea828e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7150988e5d7e48879d62ea1469e804dd"
I20260924 01:08:12.695333 31896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:12.700213 31896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2e63de21944594b76ac66d0eea828e in term 1.
I20260924 01:08:12.701140 31723 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f2e63de21944594b76ac66d0eea828e, 7150988e5d7e48879d62ea1469e804dd; no voters:
I20260924 01:08:12.701826 31941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:12.715173 31941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [term 1 LEADER]: Becoming Leader. State: Replica: 6f2e63de21944594b76ac66d0eea828e, State: Running, Role: LEADER
I20260924 01:08:12.716991 31941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.732452 31947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2e63de21944594b76ac66d0eea828e. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.733108 31947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: This master's current role is: LEADER
W20260924 01:08:12.738114 31726 tablet.cc:2369] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:08:12.742475 31959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 01:08:12.756129 31846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } attempt: 5
I20260924 01:08:12.767381 31959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:12.780148 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.720696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.8.254.125:44663)
0924 01:08:12.720968 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.720977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.737535 (+ 16558us) server_negotiation.cc:407] Connection header received
0924 01:08:12.739722 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.739731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.739852 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.739938 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.747260 (+ 7322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.748375 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.755433 (+ 7058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.756192 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.756321 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.767414 (+ 11093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.767424 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.767432 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.767480 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.771353 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.776315 (+ 4962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.776323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.776329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.776544 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.777164 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.777170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.777175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.777613 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.777887 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.779403 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.779723 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 01:08:12 dist-test-slave-5c7l krb5kdc[31692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
W20260924 01:08:12.783008 31875 tablet.cc:2369] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:08:12.784937 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.725176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.8.254.124:38053)
0924 01:08:12.725383 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.725392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.738232 (+ 12840us) server_negotiation.cc:407] Connection header received
0924 01:08:12.739225 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.739242 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.739359 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.739458 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.746885 (+ 7427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.748025 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.752208 (+ 4183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.752942 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.753078 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.774174 (+ 21096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.774182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.774187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.774267 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.778019 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.779068 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.779075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.779079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.779392 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.780987 (+ 1595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.780993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.780996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.781754 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.782276 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.784202 (+ 1926us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.784475 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 01:08:12.787738 31964 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 01:08:12.790529 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.730929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.8.254.126:35851)
0924 01:08:12.731125 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.731132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.739773 (+ 8641us) server_negotiation.cc:407] Connection header received
0924 01:08:12.740100 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.740109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.740223 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.740315 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.752232 (+ 11917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.753412 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.758073 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.758881 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.759000 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.772383 (+ 13383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.772392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.772402 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.772455 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.786418 (+ 13963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.787340 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.787347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.787351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.787513 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.787997 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.788003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.788006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.788413 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.789749 (+ 1336us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.789897 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.790132 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 01:08:12.794695 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.756797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:38053 (local address 127.0.0.1:43886)
0924 01:08:12.757188 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:12.757209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:12.757335 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:12.765631 (+ 8296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:12.765637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:12.766168 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:12.766432 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.766447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.774164 (+ 7717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.774175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.776245 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:12.776271 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.777883 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.777893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:12.778257 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.778949 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:12.778980 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:12.782757 (+ 3777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:12.789743 (+ 6986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.789749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.789753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.790440 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.792611 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:12.792618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:12.792622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:12.792820 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:12.793528 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:12.793534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:12.793754 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 01:08:12.793928 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 01:08:12.794298 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260924 01:08:12.795980 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:12.757557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.8.254.124:38053)
0924 01:08:12.765239 (+ 7682us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:12.765247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:12.765274 (+ 27us) server_negotiation.cc:407] Connection header received
0924 01:08:12.765345 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:12.765350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:12.765436 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:12.765564 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:12.767940 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.769516 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.776560 (+ 7044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:12.777617 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:12.778675 (+ 1058us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:12.783414 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:12.783423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:12.783432 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:12.783476 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:12.789559 (+ 6083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.792183 (+ 2624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.792190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.792194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.792444 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:12.793004 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:12.793010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:12.793013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:12.793375 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:12.795186 (+ 1811us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:12.795323 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 01:08:12.795564 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7516,"thread_start_us":7317,"threads_started":1}
I20260924 01:08:12.802088 31846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: Bootstrap starting.
I20260924 01:08:12.804440 31896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [term 1 FOLLOWER]: Refusing update from remote peer 6f2e63de21944594b76ac66d0eea828e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:08:12.805478 31941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:08:12.811650 31846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:12.793432 31705 master_runner.cc:186] Error getting master registration for 127.8.254.125:44663: OK, instance_id { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" instance_seqno: 1790212092025325 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 01:08:12.813665 31705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 01:08:12.814216 31846 log.cc:824] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:12.820688 31846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: No bootstrap required, opened a new log
I20260924 01:08:12.828792 31927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2e63de21944594b76ac66d0eea828e. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.829166 31927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.832093 31965 mvcc.cc:205] Tried to move back new op lower bound from 7332708732093399040 to 7332708731782189056. Current Snapshot: MvccSnapshot[applied={T|T < 7332708732093399040}]
I20260924 01:08:12.840032 31846 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.841199 31927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.841521 31927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.842926 31846 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5468755f4a6474b8f825ba969137c0b, State: Initialized, Role: FOLLOWER
I20260924 01:08:12.843556 31846 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } }
I20260924 01:08:12.849820 31947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.850121 31947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:12.847877 31971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.850732 31971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.854416 31959 catalog_manager.cc:1347] Generated new cluster ID: 52d3f9face6a4f75b7ffbdd0d9c467d2
I20260924 01:08:12.854552 31959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:12.856091 31846 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:12.859551 31820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:12.862330 31821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f2e63de21944594b76ac66d0eea828e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b5468755f4a6474b8f825ba969137c0b"
I20260924 01:08:12.863834 31820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [term 1 FOLLOWER]: Refusing update from remote peer 6f2e63de21944594b76ac66d0eea828e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:08:12.866611 31821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f2e63de21944594b76ac66d0eea828e in term 1.
I20260924 01:08:12.867527 31972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:08:12.883703 31959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:12.885531 31959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:12.904333 31959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e: Generated new TSK 0
I20260924 01:08:12.905555 31959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:12.909192 31979 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.909694 31979 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.909746 31972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.910090 31972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2e63de21944594b76ac66d0eea828e [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:12.910142 31971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2e63de21944594b76ac66d0eea828e. Latest consensus state: current_term: 1 leader_uuid: "6f2e63de21944594b76ac66d0eea828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f2e63de21944594b76ac66d0eea828e" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 35851 } } peers { permanent_uuid: "b5468755f4a6474b8f825ba969137c0b" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 44663 } } peers { permanent_uuid: "7150988e5d7e48879d62ea1469e804dd" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 38053 } } }
I20260924 01:08:12.910480 31971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:12.962074 31988 catalog_manager.cc:1259] Loaded cluster ID: 52d3f9face6a4f75b7ffbdd0d9c467d2
I20260924 01:08:12.962356 31988 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: loading cluster ID for follower catalog manager: success
I20260924 01:08:12.966452 31988 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: acquiring CA information for follower catalog manager: success
I20260924 01:08:12.969311 31988 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5468755f4a6474b8f825ba969137c0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 01:08:13.097014 31851 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.076047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.8.254.125:44663)
0924 01:08:13.076322 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.076329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.091371 (+ 15042us) server_negotiation.cc:407] Connection header received
0924 01:08:13.093338 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.093345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.093432 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.093508 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.096768 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44190: BlockingRecv error: recv got EOF from 127.0.0.1:44190 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 01:08:13.098143 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.077898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.8.254.124:38053)
0924 01:08:13.078094 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.078101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.092904 (+ 14803us) server_negotiation.cc:407] Connection header received
0924 01:08:13.094575 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.094582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.094703 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.094790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.097907 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43902: BlockingRecv error: recv got EOF from 127.0.0.1:43902 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 01:08:13.101069 31848 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.084157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.8.254.126:35851)
0924 01:08:13.084346 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.084353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.096046 (+ 11693us) server_negotiation.cc:407] Connection header received
0924 01:08:13.099135 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.099141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.099232 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.099318 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.100785 (+ 1467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33846: BlockingRecv error: recv got EOF from 127.0.0.1:33846 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 01:08:13.152786 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.137433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.8.254.125:44663)
0924 01:08:13.137635 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.137642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.144816 (+ 7174us) server_negotiation.cc:407] Connection header received
0924 01:08:13.145106 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.145114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.145335 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.145506 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.146284 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.147441 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.149561 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.150302 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.150423 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.151205 (+ 782us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.151927 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.152058 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.152308 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.152409 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 01:08:13.191826 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.181895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.8.254.124:38053)
0924 01:08:13.182146 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.182152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.189575 (+ 7423us) server_negotiation.cc:407] Connection header received
0924 01:08:13.190129 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.190135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.190314 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.190402 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.191499 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43910: BlockingRecv error: recv got EOF from 127.0.0.1:43910 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 01:08:13.242138 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.223434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.8.254.125:44663)
0924 01:08:13.223653 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.223661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.234518 (+ 10857us) server_negotiation.cc:407] Connection header received
0924 01:08:13.235084 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.235090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.235205 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.235278 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.235997 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.237218 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.239518 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.240271 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.240377 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.241138 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.241315 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.241403 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.241682 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.241815 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 01:08:13.286520 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.276785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.8.254.124:38053)
0924 01:08:13.277004 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.277011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.284509 (+ 7498us) server_negotiation.cc:407] Connection header received
0924 01:08:13.285040 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.285046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.285143 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.285219 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.286283 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43916: BlockingRecv error: recv got EOF from 127.0.0.1:43916 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 01:08:13.341696 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.329295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.8.254.125:44663)
0924 01:08:13.329495 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.329503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.336482 (+ 6979us) server_negotiation.cc:407] Connection header received
0924 01:08:13.336741 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.336746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.336824 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.336945 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.337463 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.338277 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.339771 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.340210 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.340362 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.340854 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.340999 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.341111 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.341243 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.341338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 01:08:13.358098 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.356005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.8.254.124:38053)
0924 01:08:13.356231 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.356238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.356596 (+ 358us) server_negotiation.cc:407] Connection header received
0924 01:08:13.356853 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.356859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.356949 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.357073 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.357891 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43924: BlockingRecv error: recv got EOF from 127.0.0.1:43924 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 01:08:13.373433 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.366368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.8.254.125:44663)
0924 01:08:13.366565 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.366571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.366592 (+ 21us) server_negotiation.cc:407] Connection header received
0924 01:08:13.366869 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.366877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.367015 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.367099 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.367822 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.369039 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.371138 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.371927 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.372148 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.372628 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.372774 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.372858 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.373026 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.373115 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 01:08:13.386327 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.383940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.8.254.124:38053)
0924 01:08:13.384163 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.384169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.384775 (+ 606us) server_negotiation.cc:407] Connection header received
0924 01:08:13.384984 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.384990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.385095 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.385210 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.386057 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43932: BlockingRecv error: recv got EOF from 127.0.0.1:43932 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 01:08:13.403517 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.397460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.8.254.125:44663)
0924 01:08:13.397668 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.397674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.397992 (+ 318us) server_negotiation.cc:407] Connection header received
0924 01:08:13.398149 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.398154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.398269 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.398410 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.398866 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.400030 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.401306 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.402037 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.402179 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.402631 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.402791 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.402938 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.403079 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.403170 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 01:08:13.411718 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.409292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.8.254.124:38053)
0924 01:08:13.409570 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.409576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.409928 (+ 352us) server_negotiation.cc:407] Connection header received
0924 01:08:13.410262 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.410268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.410388 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.410497 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.411465 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43946: BlockingRecv error: recv got EOF from 127.0.0.1:43946 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 01:08:13.436810 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.431318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.8.254.125:44663)
0924 01:08:13.431528 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.431537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.431906 (+ 369us) server_negotiation.cc:407] Connection header received
0924 01:08:13.432135 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.432139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.432212 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.432333 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.432757 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.433556 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.434978 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.435458 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.435601 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.436029 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.436143 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.436277 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.436434 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.436530 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 01:08:13.449922 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.447490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.8.254.124:38053)
0924 01:08:13.447741 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.447747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.448131 (+ 384us) server_negotiation.cc:407] Connection header received
0924 01:08:13.448351 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.448354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.448442 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.448524 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.449603 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43952: BlockingRecv error: recv got EOF from 127.0.0.1:43952 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 01:08:13.492992 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.486084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.8.254.125:44663)
0924 01:08:13.486384 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.486392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.486745 (+ 353us) server_negotiation.cc:407] Connection header received
0924 01:08:13.487059 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.487065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.487153 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.487214 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.487863 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.488939 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.490807 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.491446 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.491542 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.492141 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.492294 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.492353 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.492574 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.492653 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 01:08:13.504177 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.501977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.8.254.124:38053)
0924 01:08:13.502189 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.502242 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.502795 (+ 553us) server_negotiation.cc:407] Connection header received
0924 01:08:13.503047 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.503053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.503145 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.503251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.503958 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43964: BlockingRecv error: recv got EOF from 127.0.0.1:43964 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 01:08:13.577234 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.572164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.8.254.125:44663)
0924 01:08:13.572351 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.572356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.572729 (+ 373us) server_negotiation.cc:407] Connection header received
0924 01:08:13.572927 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.572931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.573006 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.573097 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.573507 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.574352 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.575762 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.576171 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.576263 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.576648 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.576749 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.576815 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.576959 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.577025 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 01:08:13.584767 31963 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:13.583055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.8.254.124:38053)
0924 01:08:13.583292 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.583299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.583517 (+ 218us) server_negotiation.cc:407] Connection header received
0924 01:08:13.583683 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.583686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.583759 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.583860 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:13.584558 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43976: BlockingRecv error: recv got EOF from 127.0.0.1:43976 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 01:08:13.654610 31938 catalog_manager.cc:1259] Loaded cluster ID: 52d3f9face6a4f75b7ffbdd0d9c467d2
I20260924 01:08:13.654774 31938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: loading cluster ID for follower catalog manager: success
I20260924 01:08:13.657671 31938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: acquiring CA information for follower catalog manager: success
I20260924 01:08:13.659672 31938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7150988e5d7e48879d62ea1469e804dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 01:08:13.721975 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.716687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.8.254.125:44663)
0924 01:08:13.716944 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.716951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.717260 (+ 309us) server_negotiation.cc:407] Connection header received
0924 01:08:13.717506 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.717511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.717582 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.717682 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.718165 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.718907 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.720259 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.720699 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.720835 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.721239 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.721355 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.721454 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.721612 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.721693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 01:08:13.733142 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.727864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.8.254.124:38053)
0924 01:08:13.728095 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.728101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.728234 (+ 133us) server_negotiation.cc:407] Connection header received
0924 01:08:13.728388 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.728391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.728545 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.728644 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.729071 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.729785 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.731084 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.731493 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.731588 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.732035 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.732542 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.732659 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.732729 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.732795 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 01:08:13.757872 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.740090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.8.254.124:38053)
0924 01:08:13.740404 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.740413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.751594 (+ 11181us) server_negotiation.cc:407] Connection header received
0924 01:08:13.751798 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.751806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.751950 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.752050 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.752837 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.753901 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.755853 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.756383 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.756505 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.757012 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.757169 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.757272 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.757483 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.757575 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260924 01:08:13.758800 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.740032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.8.254.125:44663)
0924 01:08:13.740270 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.740279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.752281 (+ 12002us) server_negotiation.cc:407] Connection header received
0924 01:08:13.752473 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.752485 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.752568 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.752672 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.753336 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.754432 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.756653 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.757382 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.757511 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.757926 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.758091 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.758195 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.758422 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.758501 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 01:08:13.759591 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:13.739640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.8.254.126:35851)
0924 01:08:13.750321 (+ 10681us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:13.750332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:13.751568 (+ 1236us) server_negotiation.cc:407] Connection header received
0924 01:08:13.751740 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:13.751750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:13.752019 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:13.752204 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:13.753057 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.753898 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.756186 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:13.756951 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:13.757084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:13.757704 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:13.758181 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:13.758315 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:13.758425 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 01:08:13.758510 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10463,"thread_start_us":10705,"threads_started":1}
I20260924 01:08:13.765561 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31705
I20260924 01:08:13.766898 31767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:13.895224 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31705
I20260924 01:08:13.917273 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31779
I20260924 01:08:13.918577 31841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:14.027354 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31779
I20260924 01:08:14.048913 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 31854
I20260924 01:08:14.050088 31916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:14.158001 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 31854
2026-09-24T01:08:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4485 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 01:08:14 dist-test-slave-5c7l krb5kdc[32102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 01:08:14 dist-test-slave-5c7l krb5kdc[32102](info): set up 2 sockets
Sep 24 01:08:14 dist-test-slave-5c7l krb5kdc[32102](info): commencing operation
krb5kdc: starting...
W20260924 01:08:15.280081 9209 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/test-user.keytab.
Sep 24 01:08:15 dist-test-slave-5c7l krb5kdc[32102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T01:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T01:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.254.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
Entry for principal kudu/127.8.254.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab.
I20260924 01:08:15.566320 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:33579
--webserver_interface=127.8.254.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:33579,127.8.254.125:41289,127.8.254.124:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 01:08:15.905771 32115 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 01:08:15.906170 32115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:15.906312 32115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:15.906392 32115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:15.913483 32115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:15.913570 32115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:15.913668 32115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:15.913724 32115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:15.913786 32115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:15.923671 32115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:33579,127.8.254.125:41289,127.8.254.124:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.126.keytab
--principal=kudu/127.8.254.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.126:33579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.254.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:15.924945 32115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:15.926260 32115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:15.938230 32120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:15.938267 32123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:15.938318 32121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:15.943910 32115 server_base.cc:1034] running on GCE node
I20260924 01:08:15.944850 32115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:15.946466 32115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:15.947742 32115 hybrid_clock.cc:648] HybridClock initialized: now 1790212095947676 us; error 39 us; skew 500 ppm
I20260924 01:08:15.948135 32115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:15 dist-test-slave-5c7l krb5kdc[32102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:15.951433 32115 init.cc:377] Logged in from keytab as kudu/127.8.254.126@KRBTEST.COM (short username kudu)
I20260924 01:08:15.953006 32115 webserver.cc:460] Webserver started at http://127.8.254.126:36733/ using document root <none> and password file <none>
I20260924 01:08:15.953809 32115 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:15.953907 32115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:15.954299 32115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:15.957487 32115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data/instance:
uuid: "8937251f7e2647509d1dae6bbabdd5c7"
format_stamp: "Formatted at 2026-09-24 01:08:15 on dist-test-slave-5c7l"
server_key: "af020b7c1d452c41d80a4df635594a50"
server_key_iv: "210736e427309b07d06d4d336aa4d054"
server_key_version: "encryptionkey@0"
I20260924 01:08:15.958442 32115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal/instance:
uuid: "8937251f7e2647509d1dae6bbabdd5c7"
format_stamp: "Formatted at 2026-09-24 01:08:15 on dist-test-slave-5c7l"
server_key: "af020b7c1d452c41d80a4df635594a50"
server_key_iv: "210736e427309b07d06d4d336aa4d054"
server_key_version: "encryptionkey@0"
I20260924 01:08:15.963917 32115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 01:08:15.967809 32130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:15.969220 32115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 01:08:15.969419 32115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal
uuid: "8937251f7e2647509d1dae6bbabdd5c7"
format_stamp: "Formatted at 2026-09-24 01:08:15 on dist-test-slave-5c7l"
server_key: "af020b7c1d452c41d80a4df635594a50"
server_key_iv: "210736e427309b07d06d4d336aa4d054"
server_key_version: "encryptionkey@0"
I20260924 01:08:15.969663 32115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:15.988224 32115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:15.992647 32115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:15.992946 32115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:16.011636 32115 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.126:33579
I20260924 01:08:16.011663 32181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.126:33579 every 8 connection(s)
I20260924 01:08:16.013098 32115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/data/info.pb
I20260924 01:08:16.016963 32182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:16.022536 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 32115
I20260924 01:08:16.022715 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-0/wal/instance
I20260924 01:08:16.023167 9209 external_mini_cluster.cc:1428] Setting key 85282156376f066bf22067dc1f73607a
I20260924 01:08:16.023736 32182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:16 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:08:16.049324 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.027013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:33579 (local address 127.0.0.1:40414)
0924 01:08:16.027955 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.028019 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:16.031077 (+ 3058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:16.032044 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:16.032053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:16.032642 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:16.032982 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.032999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.034751 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.034755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.035918 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.035931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.037275 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.037280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.037411 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.038093 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:16.038116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:16.040420 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:16.044985 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.044995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.045006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.045431 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.045989 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.045994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.046000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.046169 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.047293 (+ 1124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:16.047300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:16.047524 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 01:08:16.047736 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 01:08:16.048185 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":387,"threads_started":1}
I20260924 01:08:16.049362 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.027013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.8.254.126:33579)
0924 01:08:16.029014 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:16.029021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:16.029065 (+ 44us) server_negotiation.cc:407] Connection header received
0924 01:08:16.031506 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:16.031512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:16.031775 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:16.031934 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:16.033203 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.034598 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.036065 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.037054 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.037254 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.040668 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:16.040681 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:16.040700 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:16.040752 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:16.044827 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.045575 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.045586 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.045593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.045839 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.046333 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.046341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.046347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.047137 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:16.047379 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:16.047942 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 01:08:16.048417 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":1811,"spinlock_wait_cycles":16640,"thread_start_us":1558,"threads_started":1}
WARNING: no policy specified for kudu/127.8.254.125@KRBTEST.COM; defaulting to no policy
I20260924 01:08:16.052644 32182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:16.053809 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.053384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:34974)
0924 01:08:16.053574 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.053692 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 01:08:16.054147 32131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.125@KRBTEST.COM" created.
I20260924 01:08:16.055743 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.055413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:34984)
0924 01:08:16.055551 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.055619 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 01:08:16.056030 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:41289: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab.
Entry for principal kudu/127.8.254.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab.
I20260924 01:08:16.085367 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:41289
--webserver_interface=127.8.254.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab
--principal=kudu/127.8.254.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:33579,127.8.254.125:41289,127.8.254.124:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:08:16.105420 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } attempt: 1
I20260924 01:08:16.106791 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.106137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:34988)
0924 01:08:16.106428 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.106581 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 01:08:16.107966 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.107501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35004)
0924 01:08:16.107686 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.107789 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 01:08:16.108210 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:41289: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
I20260924 01:08:16.176481 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } attempt: 2
I20260924 01:08:16.177825 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.177157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35008)
0924 01:08:16.177477 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.177619 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 01:08:16.179121 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.178659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35010)
0924 01:08:16.178830 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.178941 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:08:16.179392 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:41289: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
I20260924 01:08:16.270678 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } attempt: 3
I20260924 01:08:16.271929 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.271384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35014)
0924 01:08:16.271633 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.271756 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 01:08:16.273123 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.272673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35018)
0924 01:08:16.272836 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.272925 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:08:16.273382 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:41289: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
I20260924 01:08:16.416663 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } attempt: 4
I20260924 01:08:16.417953 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.417321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35030)
0924 01:08:16.417654 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.417777 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260924 01:08:16.419088 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.418669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35034)
0924 01:08:16.418850 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.418937 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 01:08:16.419368 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.125:41289: Network error: Client connection negotiation failed: client connection to 127.8.254.125:41289: connect: Connection refused (error 111)
W20260924 01:08:16.427089 32189 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 01:08:16.427569 32189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:16.427672 32189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:16.427807 32189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 01:08:16.435307 32189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:16.435403 32189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:16.435521 32189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:16.435583 32189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:16.435664 32189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:16.445760 32189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:33579,127.8.254.125:41289,127.8.254.124:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.125.keytab
--principal=kudu/127.8.254.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.125:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.254.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:16.447237 32189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:16.448565 32189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 01:08:16.460512 32194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:16.460541 32197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:16.460477 32195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:16.461306 32189 server_base.cc:1034] running on GCE node
I20260924 01:08:16.462177 32189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:16.464088 32189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:16.465317 32189 hybrid_clock.cc:648] HybridClock initialized: now 1790212096465231 us; error 54 us; skew 500 ppm
I20260924 01:08:16.465687 32189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:16 dist-test-slave-5c7l krb5kdc[32102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:16.469192 32189 init.cc:377] Logged in from keytab as kudu/127.8.254.125@KRBTEST.COM (short username kudu)
I20260924 01:08:16.470849 32189 webserver.cc:460] Webserver started at http://127.8.254.125:45719/ using document root <none> and password file <none>
I20260924 01:08:16.471585 32189 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:16.471729 32189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:16.472066 32189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:16.475513 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data/instance:
uuid: "957e55736d924f86837bc6b616feb4e6"
format_stamp: "Formatted at 2026-09-24 01:08:16 on dist-test-slave-5c7l"
server_key: "6bce22c788d67d6ba202e8c118a06571"
server_key_iv: "5332d92384783d1efd2608628d8185c2"
server_key_version: "encryptionkey@0"
I20260924 01:08:16.476337 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal/instance:
uuid: "957e55736d924f86837bc6b616feb4e6"
format_stamp: "Formatted at 2026-09-24 01:08:16 on dist-test-slave-5c7l"
server_key: "6bce22c788d67d6ba202e8c118a06571"
server_key_iv: "5332d92384783d1efd2608628d8185c2"
server_key_version: "encryptionkey@0"
I20260924 01:08:16.482059 32189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 01:08:16.485635 32204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:16.487183 32189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 01:08:16.487381 32189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal
uuid: "957e55736d924f86837bc6b616feb4e6"
format_stamp: "Formatted at 2026-09-24 01:08:16 on dist-test-slave-5c7l"
server_key: "6bce22c788d67d6ba202e8c118a06571"
server_key_iv: "5332d92384783d1efd2608628d8185c2"
server_key_version: "encryptionkey@0"
I20260924 01:08:16.487617 32189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:16.503063 32189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:16.507649 32189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:16.507953 32189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:16.526839 32189 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.125:41289
I20260924 01:08:16.526855 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.125:41289 every 8 connection(s)
I20260924 01:08:16.528321 32189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/data/info.pb
I20260924 01:08:16.531597 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 32189
I20260924 01:08:16.531810 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-1/wal/instance
I20260924 01:08:16.532259 9209 external_mini_cluster.cc:1428] Setting key 41e408eda2fc57418828c2eb328a4f5b
I20260924 01:08:16.532338 32256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:16.539402 32256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:16 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.254.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.254.124@KRBTEST.COM" created.
I20260924 01:08:16.563197 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.542835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.8.254.126:33579)
0924 01:08:16.543084 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:16.543094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:16.544339 (+ 1245us) server_negotiation.cc:407] Connection header received
0924 01:08:16.546335 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:16.546342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:16.546441 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:16.546565 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:16.548246 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.549416 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.551239 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.552012 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.552167 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.556185 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:16.556194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:16.556202 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:16.556254 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:16.559588 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.560586 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.560595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.560601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.560751 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.561323 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.561327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.561329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.561613 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:16.561727 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:16.562785 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0924 01:08:16.562971 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 01:08:16.564961 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.543300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:33579 (local address 127.0.0.1:40426)
0924 01:08:16.544129 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.544189 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:16.546003 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:16.546907 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:16.546922 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:16.547470 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:16.548053 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.548072 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.549616 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.549621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.551066 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.551079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.552245 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.552250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.552451 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.553220 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:16.553254 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:16.555904 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:16.559787 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.559794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.559809 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.560457 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.560907 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.560912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.560916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.561205 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.561755 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:16.561767 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:16.562362 (+ 595us) client_negotiation.cc:769] Sending connection context
0924 01:08:16.562702 (+ 340us) client_negotiation.cc:240] Negotiation successful
0924 01:08:16.563426 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":375,"threads_started":1}
I20260924 01:08:16.566810 32256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:16 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:08:16.582697 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.567610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35048)
0924 01:08:16.567947 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.567969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:16.568078 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:16.569204 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:16.569211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:16.569711 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:16.569912 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.569925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.571813 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.571821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.572672 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.572683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.573677 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.573683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.573908 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.574558 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:16.574575 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:16.576807 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:16.580133 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.580140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.580147 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.580503 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.580913 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.580917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.580921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.581040 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.581659 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:16.581670 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:16.581824 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 01:08:16.581966 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 01:08:16.582273 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":3}
I20260924 01:08:16.582744 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.567827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.8.254.125:41289)
0924 01:08:16.568675 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:16.568685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:16.568718 (+ 33us) server_negotiation.cc:407] Connection header received
0924 01:08:16.568774 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:16.568785 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:16.569028 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:16.569202 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:16.570109 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.571674 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.572836 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.573512 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.573664 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.577022 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:16.577034 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:16.577048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:16.577076 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:16.579999 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.580635 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.580650 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.580652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.580779 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.581154 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.581160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.581163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.581526 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:16.581739 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:16.582118 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 01:08:16.582368 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":647,"thread_start_us":474,"threads_started":1}
I20260924 01:08:16.585672 32256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:16.586886 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.586406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41572)
0924 01:08:16.586578 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.586688 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 01:08:16.587177 32206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
I20260924 01:08:16.589066 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.588674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41588)
0924 01:08:16.588871 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.588958 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 01:08:16.589316 32256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.254.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab.
Entry for principal kudu/127.8.254.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab.
I20260924 01:08:16.595613 9209 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
/tmp/dist-test-taskpTDkWM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:45707
--webserver_interface=127.8.254.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab
--principal=kudu/127.8.254.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.254.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.254.126:33579,127.8.254.125:41289,127.8.254.124:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 01:08:16.638652 32256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 1
I20260924 01:08:16.639963 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.639411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41590)
0924 01:08:16.639637 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.639771 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 01:08:16.641041 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.640625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41602)
0924 01:08:16.640794 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.640898 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 01:08:16.641265 32256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
I20260924 01:08:16.709527 32256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 2
I20260924 01:08:16.710834 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.710314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41616)
0924 01:08:16.710530 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.710647 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 01:08:16.711855 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.711484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41618)
0924 01:08:16.711634 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.711720 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 01:08:16.712064 32256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
I20260924 01:08:16.718628 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } attempt: 5
Sep 24 01:08:16 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:08:16.732892 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.719276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35056)
0924 01:08:16.719552 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.719568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:16.719678 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:16.720052 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:16.720055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:16.720433 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:16.720609 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.720617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.721937 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.721941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.722862 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:16.722875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.723824 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.723828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:16.723992 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.724635 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:16.724654 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:16.726810 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:16.730545 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.730549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.730552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.730970 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.731390 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:16.731393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:16.731396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:16.731524 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:16.732125 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:16.732128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:16.732263 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 01:08:16.732388 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 01:08:16.732630 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 01:08:16.733065 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.719370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.8.254.125:41289)
0924 01:08:16.719543 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:16.719550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:16.719621 (+ 71us) server_negotiation.cc:407] Connection header received
0924 01:08:16.719796 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:16.719801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:16.719896 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:16.720014 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:16.720753 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.721809 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.723011 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:16.723675 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:16.723799 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:16.727028 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:16.727036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:16.727041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:16.727085 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:16.730384 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.731096 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.731102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.731106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.731270 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:16.731629 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:16.731635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:16.731638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:16.732010 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:16.732172 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:16.732531 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 01:08:16.732769 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 01:08:16.734556 32182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:16.735654 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.735259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41620)
0924 01:08:16.735434 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.735513 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 01:08:16.736613 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.736237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41632)
0924 01:08:16.736421 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.736507 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 01:08:16.736821 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
I20260924 01:08:16.788050 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 1
I20260924 01:08:16.789193 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.788713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41642)
0924 01:08:16.788930 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.789029 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 01:08:16.790254 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.789852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41650)
0924 01:08:16.790014 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.790098 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 01:08:16.790498 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
I20260924 01:08:16.803329 32256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 3
I20260924 01:08:16.804761 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.804083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41660)
0924 01:08:16.804368 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.804517 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 01:08:16.806090 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.805521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41672)
0924 01:08:16.805800 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.805921 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260924 01:08:16.806353 32256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
I20260924 01:08:16.858783 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 2
I20260924 01:08:16.860078 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.859506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41676)
0924 01:08:16.859815 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.859915 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 01:08:16.861285 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.860917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41692)
0924 01:08:16.861089 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.861171 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 01:08:16.861606 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
W20260924 01:08:16.943531 32263 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 01:08:16.943931 32263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 01:08:16.944049 32263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 01:08:16.944195 32263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 01:08:16.949570 32256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 4
I20260924 01:08:16.950942 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.950436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41696)
0924 01:08:16.950692 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.950787 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 01:08:16.952032 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.951682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41700)
0924 01:08:16.951856 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.951927 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 01:08:16.952294 32256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
W20260924 01:08:16.952910 32263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 01:08:16.952991 32263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 01:08:16.953086 32263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 01:08:16.953145 32263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 01:08:16.953228 32263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 01:08:16.963768 32263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.254.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.254.126:33579,127.8.254.125:41289,127.8.254.124:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/krb5kdc/kudu_127.8.254.124.keytab
--principal=kudu/127.8.254.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.254.124:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.254.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 01:08:16.965206 32263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 01:08:16.966727 32263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 01:08:16.967890 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 3
I20260924 01:08:16.969076 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.968600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41702)
0924 01:08:16.968827 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.968915 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 01:08:16.970112 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:16.969755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41716)
0924 01:08:16.969946 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:16.970012 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 01:08:16.970391 32182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.254.124:45707: Network error: Client connection negotiation failed: client connection to 127.8.254.124:45707: connect: Connection refused (error 111)
W20260924 01:08:16.979890 32268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:16.979930 32269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 01:08:16.979831 32271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 01:08:16.981086 32263 server_base.cc:1034] running on GCE node
I20260924 01:08:16.981907 32263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 01:08:16.983556 32263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 01:08:16.984776 32263 hybrid_clock.cc:648] HybridClock initialized: now 1790212096984710 us; error 37 us; skew 500 ppm
I20260924 01:08:16.985195 32263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 01:08:16 dist-test-slave-5c7l krb5kdc[32102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 01:08:16.988240 32263 init.cc:377] Logged in from keytab as kudu/127.8.254.124@KRBTEST.COM (short username kudu)
I20260924 01:08:16.989688 32263 webserver.cc:460] Webserver started at http://127.8.254.124:34889/ using document root <none> and password file <none>
I20260924 01:08:16.990497 32263 fs_manager.cc:359] Metadata directory not provided
I20260924 01:08:16.990634 32263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 01:08:16.991000 32263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 01:08:16.994136 32263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data/instance:
uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c"
format_stamp: "Formatted at 2026-09-24 01:08:16 on dist-test-slave-5c7l"
server_key: "08dfdf43bc17069bff2b4a718015284b"
server_key_iv: "f8f063d35c4e51c9ff241cceb691ddbb"
server_key_version: "encryptionkey@0"
I20260924 01:08:16.995023 32263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal/instance:
uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c"
format_stamp: "Formatted at 2026-09-24 01:08:16 on dist-test-slave-5c7l"
server_key: "08dfdf43bc17069bff2b4a718015284b"
server_key_iv: "f8f063d35c4e51c9ff241cceb691ddbb"
server_key_version: "encryptionkey@0"
I20260924 01:08:17.000524 32263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 01:08:17.004160 32278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 01:08:17.005604 32263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 01:08:17.005811 32263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data,/tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal
uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c"
format_stamp: "Formatted at 2026-09-24 01:08:16 on dist-test-slave-5c7l"
server_key: "08dfdf43bc17069bff2b4a718015284b"
server_key_iv: "f8f063d35c4e51c9ff241cceb691ddbb"
server_key_version: "encryptionkey@0"
I20260924 01:08:17.006044 32263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 01:08:17.028908 32263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 01:08:17.033314 32263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 01:08:17.033628 32263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 01:08:17.052626 32263 rpc_server.cc:307] RPC server started. Bound to: 127.8.254.124:45707
I20260924 01:08:17.052649 32329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.254.124:45707 every 8 connection(s)
I20260924 01:08:17.054061 32263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/data/info.pb
I20260924 01:08:17.057832 32330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 01:08:17.062816 9209 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu as pid 32263
I20260924 01:08:17.063004 9209 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpTDkWM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790211722208042-9209-0/minicluster-data/master-2/wal/instance
I20260924 01:08:17.063536 9209 external_mini_cluster.cc:1428] Setting key 22f5f569963d2cb1d501605baa3f0261
I20260924 01:08:17.065068 32330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
I20260924 01:08:17.084507 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.067836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.8.254.126:33579)
0924 01:08:17.068444 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.068453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.068923 (+ 470us) server_negotiation.cc:407] Connection header received
0924 01:08:17.069308 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.069314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.069395 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.069511 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.070807 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.071560 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.073130 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.073734 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.073862 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.077895 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.077900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.077903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.077937 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.081033 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.081999 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.082003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.082005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.082130 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.082640 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.082648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.082653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.082988 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.083166 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.083943 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.084110 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":358,"threads_started":1}
I20260924 01:08:17.085637 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.067866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.126:33579 (local address 127.0.0.1:40430)
0924 01:08:17.068781 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:17.068815 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:17.069080 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:17.069572 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:17.069578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:17.070260 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:17.070645 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.070663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.071719 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.071731 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.072994 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.073006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.074023 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.074030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.074221 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.074949 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:17.074988 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:17.077694 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:17.081176 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.081193 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.081207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.081877 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.082270 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.082278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.082283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.082518 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.083131 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:17.083150 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:17.083581 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 01:08:17.083851 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 01:08:17.084388 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":26,"thread_start_us":563,"threads_started":1}
I20260924 01:08:17.085915 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.065405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.8.254.124:45707)
0924 01:08:17.066265 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.066282 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.066329 (+ 47us) server_negotiation.cc:407] Connection header received
0924 01:08:17.068653 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.068679 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.069124 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.069351 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.070900 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.072151 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.073477 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.074306 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.074409 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.079077 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.079086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.079098 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.079131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.082430 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.083253 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.083260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.083263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.083405 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.083743 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.083747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.083750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.084173 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.084450 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.085334 (+ 884us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.085514 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":600,"thread_start_us":388,"threads_started":1}
I20260924 01:08:17.087206 32330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.126@KRBTEST.COM
I20260924 01:08:17.102470 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.087976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.8.254.125:41289)
0924 01:08:17.088261 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.088265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.088505 (+ 240us) server_negotiation.cc:407] Connection header received
0924 01:08:17.088648 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.088654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.088745 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.088861 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.089604 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.090705 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.092132 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.092808 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.092948 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.096064 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.096073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.096080 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.096128 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.099589 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.100444 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.100449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.100452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.100627 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.101010 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.101015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.101018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.101397 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.101550 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.101933 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.102089 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260924 01:08:17.102514 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.087998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.125:41289 (local address 127.0.0.1:35064)
0924 01:08:17.088390 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:17.088410 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:17.088518 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:17.089007 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:17.089010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:17.089313 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:17.089448 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.089455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.090840 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.090846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.091989 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.092001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.092949 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.092953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.093102 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.093656 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:17.093680 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:17.095856 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:17.099782 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.099786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.099789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.100323 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.100756 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.100760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.100762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.100898 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.101527 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:17.101530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:17.101668 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 01:08:17.101825 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 01:08:17.102111 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260924 01:08:17.103689 32330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } has no permanent_uuid. Determining permanent_uuid...
I20260924 01:08:17.104871 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.088744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.8.254.126:33579)
0924 01:08:17.088902 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.088910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.088930 (+ 20us) server_negotiation.cc:407] Connection header received
0924 01:08:17.088978 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.088983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.089058 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.089164 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.090530 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.091520 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.093265 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.093893 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.094026 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.098180 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.098188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.098212 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.098260 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.101673 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.102480 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.102489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.102495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.102651 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.103191 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.103196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.103199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.103585 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.103813 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.104330 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.104569 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.124@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.254.125@KRBTEST.COM
I20260924 01:08:17.117435 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.104647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.8.254.124:45707)
0924 01:08:17.104793 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.104798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.104812 (+ 14us) server_negotiation.cc:407] Connection header received
0924 01:08:17.104865 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.104869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.104916 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.105014 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.105859 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.106907 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.108122 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.108619 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.108714 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.112030 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.112036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.112041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.112080 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.114772 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.115428 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.115435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.115440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.115603 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.115958 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.115963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.115966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.116294 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.116412 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.116865 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.117192 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":36}
I20260924 01:08:17.117511 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.104432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41738)
0924 01:08:17.104587 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:17.104607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:17.104699 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:17.105031 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:17.105035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:17.105486 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:17.105677 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.105688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.107033 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.107038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.107982 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.107994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.108766 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.108773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.109004 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.109610 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:17.109626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:17.111861 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:17.114898 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.114902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.114905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.115307 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.115723 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.115726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.115728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.115847 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.116411 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:17.116417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:17.116584 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 01:08:17.116744 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 01:08:17.117058 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 01:08:17.121797 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.106750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.8.254.125:41289)
0924 01:08:17.106929 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.106933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.106947 (+ 14us) server_negotiation.cc:407] Connection header received
0924 01:08:17.107011 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.107015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.107086 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.107184 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.108336 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.109288 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.110727 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.111385 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.111508 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.116128 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.116133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.116136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.116166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.119203 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.120014 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.120018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.120020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.120143 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.120559 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.120562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.120564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.120806 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.120987 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.121362 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.121516 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 01:08:17.124074 32330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: Bootstrap starting.
I20260924 01:08:17.126543 9209 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 01:08:17.127826 32330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:17.128994 32330 log.cc:824] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:17.128844 32157 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8937251f7e2647509d1dae6bbabdd5c7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40446
I20260924 01:08:17.131850 32330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: No bootstrap required, opened a new log
I20260924 01:08:17.139250 32330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.139623 32330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd3d4a0d52040d08bf7d2dd9a8c921c, State: Initialized, Role: FOLLOWER
I20260924 01:08:17.140081 32330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.141175 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.141552 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.142287 32330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:17.147596 32182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 4
W20260924 01:08:17.152727 32348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:08:17.152854 32348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212095, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.126@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
I20260924 01:08:17.150262 32263 master_runner.cc:186] Error getting master registration for 127.8.254.126:33579: OK, instance_id { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" instance_seqno: 1790212096006411 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 01:08:17.162945 32263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 01:08:17.165598 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.148430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41752)
0924 01:08:17.148655 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:17.148679 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:17.148823 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:17.149590 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:17.149595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:17.150165 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:17.150400 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.150412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.152020 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.152028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.153483 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.153501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.154690 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.154701 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.154933 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.155647 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:17.155671 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:17.158581 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:17.162693 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.162700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.162704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.163338 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.163754 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.163760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.163764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.163915 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.164547 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:17.164553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:17.164728 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 01:08:17.164885 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 01:08:17.165201 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 01:08:17.165598 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.148957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.8.254.124:45707)
0924 01:08:17.149124 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.149130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.149148 (+ 18us) server_negotiation.cc:407] Connection header received
0924 01:08:17.149213 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.149217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.149302 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.149403 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.150572 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.151787 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.153668 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.154485 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.154628 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.158829 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.158834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.158838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.158871 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.162514 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.163486 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.163492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.163495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.163626 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.164030 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.164035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.164039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.164399 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.164575 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.164982 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.165200 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 01:08:17.171064 32182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: Bootstrap starting.
I20260924 01:08:17.175478 32182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: Neither blocks nor log segments found. Creating new log.
I20260924 01:08:17.176872 32182 log.cc:824] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:17.180378 32182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: No bootstrap required, opened a new log
I20260924 01:08:17.185563 32182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.185917 32182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8937251f7e2647509d1dae6bbabdd5c7, State: Initialized, Role: FOLLOWER
I20260924 01:08:17.186592 32182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.187625 32351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.188040 32351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.188853 32182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:17.188907 32157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 01:08:17.189082 32157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:17.193436 32157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.197587 32305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8937251f7e2647509d1dae6bbabdd5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c"
I20260924 01:08:17.198247 32305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:17.201848 32305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8937251f7e2647509d1dae6bbabdd5c7 in term 1.
I20260924 01:08:17.202636 32132 leader_election.cc:302] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8937251f7e2647509d1dae6bbabdd5c7, 8cd3d4a0d52040d08bf7d2dd9a8c921c; no voters:
I20260924 01:08:17.203104 32351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 01:08:17.210234 32157 leader_election.cc:288] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [CANDIDATE]: Term 1 election: Requested vote from peers 957e55736d924f86837bc6b616feb4e6 (127.8.254.125:41289), 8cd3d4a0d52040d08bf7d2dd9a8c921c (127.8.254.124:45707)
I20260924 01:08:17.224852 32351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8937251f7e2647509d1dae6bbabdd5c7, State: Running, Role: LEADER
I20260924 01:08:17.226480 32351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.202570 32115 master_runner.cc:186] Error getting master registration for 127.8.254.125:41289: OK, instance_id { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" instance_seqno: 1790212096521287 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 01:08:17.228855 32115 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 01:08:17.236172 32362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 01:08:17.236335 32362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 01:08:17.241818 32364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8937251f7e2647509d1dae6bbabdd5c7. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.242355 32364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:17.243829 32372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 01:08:17.246047 32136 tablet.cc:2369] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:08:17.249445 32372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 01:08:17.251511 32256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } attempt: 5
Sep 24 01:08:17 dist-test-slave-5c7l krb5kdc[32102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790212096, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.254.125@KRBTEST.COM for kudu/127.8.254.124@KRBTEST.COM
I20260924 01:08:17.271394 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.252313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.254.124:45707 (local address 127.0.0.1:41766)
0924 01:08:17.252869 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 01:08:17.252894 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 01:08:17.253027 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 01:08:17.253621 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 01:08:17.253627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 01:08:17.254166 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 01:08:17.254415 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.254427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.255949 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.255956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.257347 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 01:08:17.257363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.258916 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.258925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 01:08:17.259211 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.259959 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 01:08:17.259985 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 01:08:17.263065 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 01:08:17.267969 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.267977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.267984 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.268661 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.269192 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 01:08:17.269197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 01:08:17.269202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 01:08:17.269404 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 01:08:17.270176 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 01:08:17.270185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 01:08:17.270394 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 01:08:17.270566 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 01:08:17.270917 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386}
I20260924 01:08:17.273656 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.252804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.8.254.124:45707)
0924 01:08:17.252994 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.253000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.253019 (+ 19us) server_negotiation.cc:407] Connection header received
0924 01:08:17.253170 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.253177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.253295 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.253402 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.254604 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.255772 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.257551 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.258514 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.258665 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.263346 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.263355 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.263361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.263408 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.267708 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.268830 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.268839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.268844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.269043 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.269553 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.269560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.269563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.269952 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.270109 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.271823 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.272106 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 01:08:17.280393 32256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: Bootstrap starting.
I20260924 01:08:17.285324 32256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: Neither blocks nor log segments found. Creating new log.
W20260924 01:08:17.285862 32284 tablet.cc:2369] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 01:08:17.286717 32256 log.cc:824] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: Log is configured to *not* fsync() on all Append() calls
I20260924 01:08:17.303586 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.240667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.8.254.125:41289)
0924 01:08:17.242475 (+ 1808us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.242481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.254694 (+ 12213us) server_negotiation.cc:407] Connection header received
0924 01:08:17.268658 (+ 13964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.268665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.268756 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.268830 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.277745 (+ 8915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.278928 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.283589 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.284288 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.284401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.290899 (+ 6498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.290906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.290912 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.290955 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.294640 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.301062 (+ 6422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.301070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.301075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.301249 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.301714 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.301719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.301722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.302108 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.302367 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.302955 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.303223 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609}
I20260924 01:08:17.307787 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.255183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.8.254.124:45707)
0924 01:08:17.272793 (+ 17610us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.272799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.272823 (+ 24us) server_negotiation.cc:407] Connection header received
0924 01:08:17.272873 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.272882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.272970 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.273075 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.276208 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.277541 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.282479 (+ 4938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.283212 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.283339 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.296074 (+ 12735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.296082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.296087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.296131 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.300115 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.304591 (+ 4476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.304609 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.304613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.304811 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.305307 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.305312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.305316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.305736 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.306007 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.307111 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.307349 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17469,"thread_start_us":19278,"threads_started":1}
I20260924 01:08:17.310897 32305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [term 1 FOLLOWER]: Refusing update from remote peer 8937251f7e2647509d1dae6bbabdd5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:08:17.311951 32351 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:08:17.313019 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:17.242246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.8.254.126:33579)
0924 01:08:17.242862 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.242870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.251511 (+ 8641us) server_negotiation.cc:407] Connection header received
0924 01:08:17.252045 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.252054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.252155 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.252238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.288497 (+ 36259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.289759 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.291994 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:17.292783 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:17.292902 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:17.300630 (+ 7728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 01:08:17.300637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 01:08:17.300642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 01:08:17.300693 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 01:08:17.304337 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.308755 (+ 4418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.308761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.308765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.310145 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 01:08:17.310724 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 01:08:17.310730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 01:08:17.310735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 01:08:17.311218 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 01:08:17.311557 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:17.312272 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 01:08:17.312520 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260924 01:08:17.315286 32256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: No bootstrap required, opened a new log
I20260924 01:08:17.335918 32256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.336375 32256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957e55736d924f86837bc6b616feb4e6, State: Initialized, Role: FOLLOWER
I20260924 01:08:17.338979 32256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } }
I20260924 01:08:17.341087 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8937251f7e2647509d1dae6bbabdd5c7. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.341420 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.347101 32374 mvcc.cc:205] Tried to move back new op lower bound from 7332708750530994176 to 7332708750253891584. Current Snapshot: MvccSnapshot[applied={T|T < 7332708750530994176}]
I20260924 01:08:17.348258 32351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.348584 32351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:17.350709 32377 mvcc.cc:205] Tried to move back new op lower bound from 7332708750530994176 to 7332708750253891584. Current Snapshot: MvccSnapshot[applied={T|T < 7332708750530994176}]
I20260924 01:08:17.357595 32337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.357872 32337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.358865 32372 catalog_manager.cc:1347] Generated new cluster ID: 9f3f9f6a2bfb414199881e51cce6b8f1
I20260924 01:08:17.358973 32372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 01:08:17.362785 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.363222 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.365733 32230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 01:08:17.366319 32231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8937251f7e2647509d1dae6bbabdd5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "957e55736d924f86837bc6b616feb4e6"
I20260924 01:08:17.369715 32230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [term 1 FOLLOWER]: Refusing update from remote peer 8937251f7e2647509d1dae6bbabdd5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 01:08:17.370416 32256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 01:08:17.371083 32351 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 01:08:17.373303 32231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8937251f7e2647509d1dae6bbabdd5c7 in term 1.
I20260924 01:08:17.391328 32372 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 01:08:17.393340 32372 catalog_manager.cc:1504] Loading token signing keys...
I20260924 01:08:17.400322 32364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.400720 32364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7 [sys.catalog]: This master's current role is: LEADER
I20260924 01:08:17.401894 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8937251f7e2647509d1dae6bbabdd5c7. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.402243 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.402525 32385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8937251f7e2647509d1dae6bbabdd5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937251f7e2647509d1dae6bbabdd5c7" member_type: VOTER last_known_addr { host: "127.8.254.126" port: 33579 } } peers { permanent_uuid: "957e55736d924f86837bc6b616feb4e6" member_type: VOTER last_known_addr { host: "127.8.254.125" port: 41289 } } peers { permanent_uuid: "8cd3d4a0d52040d08bf7d2dd9a8c921c" member_type: VOTER last_known_addr { host: "127.8.254.124" port: 45707 } } }
I20260924 01:08:17.402905 32385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 01:08:17.417697 32372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8937251f7e2647509d1dae6bbabdd5c7: Generated new TSK 0
I20260924 01:08:17.418818 32372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 01:08:17.442611 32397 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 01:08:17.513981 32334 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.500066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.8.254.126:33579)
0924 01:08:17.500477 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.500484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.511811 (+ 11327us) server_negotiation.cc:407] Connection header received
0924 01:08:17.512123 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.512130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.512221 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.512298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.513572 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40466: BlockingRecv error: recv got EOF from 127.0.0.1:40466 (error 108)
Metrics: {"server-negotiator.queue_time_us":272}
W20260924 01:08:17.516360 32373 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.510853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.8.254.124:45707)
0924 01:08:17.511053 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.511059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.514675 (+ 3616us) server_negotiation.cc:407] Connection header received
0924 01:08:17.514932 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.514938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.515016 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.515086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.516121 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41782: BlockingRecv error: recv got EOF from 127.0.0.1:41782 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 01:08:17.516439 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.500260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.8.254.125:41289)
0924 01:08:17.500607 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.500613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.513741 (+ 13128us) server_negotiation.cc:407] Connection header received
0924 01:08:17.514876 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.514883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.514968 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.515045 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.516218 (+ 1173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35092: BlockingRecv error: recv got EOF from 127.0.0.1:35092 (error 108)
Metrics: {"server-negotiator.queue_time_us":242}
W20260924 01:08:17.524283 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.520973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.8.254.125:41289)
0924 01:08:17.521167 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.521173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.522722 (+ 1549us) server_negotiation.cc:407] Connection header received
0924 01:08:17.522946 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.522952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.523092 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.523201 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.524008 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 01:08:17.562835 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.560474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.8.254.125:41289)
0924 01:08:17.560681 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.560689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.561241 (+ 552us) server_negotiation.cc:407] Connection header received
0924 01:08:17.561451 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.561457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.561556 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.561676 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.562607 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35100: BlockingRecv error: recv got EOF from 127.0.0.1:35100 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 01:08:17.572429 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.568137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.8.254.125:41289)
0924 01:08:17.568304 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.568310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.570767 (+ 2457us) server_negotiation.cc:407] Connection header received
0924 01:08:17.570985 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.570991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.571085 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.571196 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.572175 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35102: BlockingRecv error: recv got EOF from 127.0.0.1:35102 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 01:08:17.582620 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.580332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.8.254.125:41289)
0924 01:08:17.580546 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.580551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.581055 (+ 504us) server_negotiation.cc:407] Connection header received
0924 01:08:17.581258 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.581264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.581363 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.581474 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.582384 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35110: BlockingRecv error: recv got EOF from 127.0.0.1:35110 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 01:08:17.596451 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.594071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.8.254.125:41289)
0924 01:08:17.594309 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.594315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.594878 (+ 563us) server_negotiation.cc:407] Connection header received
0924 01:08:17.595174 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.595180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.595282 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.595392 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.596215 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35124: BlockingRecv error: recv got EOF from 127.0.0.1:35124 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 01:08:17.618728 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.616265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.8.254.125:41289)
0924 01:08:17.616465 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.616471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.617053 (+ 582us) server_negotiation.cc:407] Connection header received
0924 01:08:17.617274 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.617280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.617389 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.617507 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.618462 (+ 955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35128: BlockingRecv error: recv got EOF from 127.0.0.1:35128 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 01:08:17.657373 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.654916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.8.254.125:41289)
0924 01:08:17.655109 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.655125 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.655687 (+ 562us) server_negotiation.cc:407] Connection header received
0924 01:08:17.655917 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.655923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.656028 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.656142 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.657095 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35140: BlockingRecv error: recv got EOF from 127.0.0.1:35140 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 01:08:17.729488 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.727249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.8.254.125:41289)
0924 01:08:17.727510 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.727516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.727871 (+ 355us) server_negotiation.cc:407] Connection header received
0924 01:08:17.728104 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.728108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.728200 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.728288 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.729253 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35156: BlockingRecv error: recv got EOF from 127.0.0.1:35156 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 01:08:17.863361 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:17.861369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.8.254.125:41289)
0924 01:08:17.861611 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:17.861618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:17.861867 (+ 249us) server_negotiation.cc:407] Connection header received
0924 01:08:17.862112 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:17.862125 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:17.862262 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:17.862380 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:17.863107 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35160: BlockingRecv error: recv got EOF from 127.0.0.1:35160 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 01:08:18.124866 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 01:08:18.122899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.8.254.125:41289)
0924 01:08:18.123143 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:18.123151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:18.123400 (+ 249us) server_negotiation.cc:407] Connection header received
0924 01:08:18.123639 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:18.123645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:18.123789 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:18.123901 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 01:08:18.124652 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35166: BlockingRecv error: recv got EOF from 127.0.0.1:35166 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 01:08:18.154904 32348 catalog_manager.cc:1259] Loaded cluster ID: 9f3f9f6a2bfb414199881e51cce6b8f1
I20260924 01:08:18.155087 32348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: loading cluster ID for follower catalog manager: success
I20260924 01:08:18.157680 32348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: acquiring CA information for follower catalog manager: success
I20260924 01:08:18.159603 32348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8cd3d4a0d52040d08bf7d2dd9a8c921c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 01:08:18.447297 32397 catalog_manager.cc:1259] Loaded cluster ID: 9f3f9f6a2bfb414199881e51cce6b8f1
I20260924 01:08:18.447455 32397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: loading cluster ID for follower catalog manager: success
I20260924 01:08:18.450089 32397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: acquiring CA information for follower catalog manager: success
I20260924 01:08:18.451913 32397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 957e55736d924f86837bc6b616feb4e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 01:08:18.650079 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:18.642409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.8.254.125:41289)
0924 01:08:18.643238 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:18.643245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:18.643262 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:08:18.643335 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:18.643345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:18.643500 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:18.643659 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:18.644216 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.645299 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.647466 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.647945 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.648055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:18.648687 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:18.649310 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:18.649422 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:18.649572 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 01:08:18.649659 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":523,"threads_started":1}
I20260924 01:08:18.663798 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:18.656829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.8.254.124:45707)
0924 01:08:18.658531 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:18.658535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:18.658552 (+ 17us) server_negotiation.cc:407] Connection header received
0924 01:08:18.658645 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:18.658649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:18.658814 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:18.658993 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:18.659429 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.660120 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.661469 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.661858 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.661994 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:18.662606 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:18.663144 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:18.663325 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:18.663423 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 01:08:18.663511 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1605,"thread_start_us":1515,"threads_started":1}
I20260924 01:08:18.684468 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:18.669925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.8.254.125:41289)
0924 01:08:18.670477 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:18.670482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:18.678924 (+ 8442us) server_negotiation.cc:407] Connection header received
0924 01:08:18.679155 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:18.679161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:18.679245 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:18.679352 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:18.679936 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.680671 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.682075 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.682803 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.682952 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:18.683574 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:18.683703 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:18.683771 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:18.684037 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 01:08:18.684128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260924 01:08:18.687422 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:18.670149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.8.254.126:33579)
0924 01:08:18.678628 (+ 8479us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:18.678649 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:18.678683 (+ 34us) server_negotiation.cc:407] Connection header received
0924 01:08:18.678843 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:18.678851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:18.679150 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:18.680334 (+ 1184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:18.680566 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.681970 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.684148 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.684848 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.685653 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:18.685859 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:18.686353 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:18.686974 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:18.687031 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 01:08:18.687117 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8302,"thread_start_us":8157,"threads_started":1}
I20260924 01:08:18.692224 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 01:08:18.670849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.8.254.124:45707)
0924 01:08:18.671081 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 01:08:18.671088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 01:08:18.685544 (+ 14456us) server_negotiation.cc:407] Connection header received
0924 01:08:18.685856 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 01:08:18.685862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 01:08:18.685960 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 01:08:18.686025 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 01:08:18.686621 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.687672 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.690255 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 01:08:18.690834 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 01:08:18.690965 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 01:08:18.691431 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 01:08:18.691566 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 01:08:18.691655 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 01:08:18.691819 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 01:08:18.691935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 01:08:18.692370 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 32115
I20260924 01:08:18.693410 32177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:18.825904 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 32115
I20260924 01:08:18.847668 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 32189
I20260924 01:08:18.848754 32251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:18.954427 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 32189
I20260924 01:08:18.975211 9209 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu pid 32263
I20260924 01:08:18.976379 32325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 01:08:19.083436 9209 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpTDkWM/build/asan/bin/kudu with pid 32263
2026-09-24T01:08:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4920 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9406 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (376830 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST